[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785997581049705-21554-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 06:26:22 dist-test-slave-4tdj krb5kdc[21563](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 06:26:22 dist-test-slave-4tdj krb5kdc[21563](info): set up 2 sockets
Aug 06 06:26:22 dist-test-slave-4tdj krb5kdc[21563](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260806 06:26:23.122110 21554 mini_kdc.cc:121] Time spent starting KDC: real 2.054s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785997581049705-21554-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785997581049705-21554-0/krb5kdc/test-user.keytab.
Aug 06 06:26:23 dist-test-slave-4tdj krb5kdc[21563](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997583, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T06:26:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T06:26:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.12.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.12.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.12.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.190.keytab.
Entry for principal kudu/127.21.12.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.190.keytab.
I20260806 06:26:23.237136 21554 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHhLhT/build/release/bin/kudu
/tmp/dist-test-taskFHhLhT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785997581049705-21554-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785997581049705-21554-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785997581049705-21554-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785997581049705-21554-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.12.190:34787
--webserver_interface=127.21.12.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.190.keytab
--principal=kudu/127.21.12.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.148:36221
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.12.190:34787
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785997581049705-21554-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785997581049705-21554-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785997581049705-21554-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 06:26:23.318087 21577 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 06:26:23.318325 21577 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 06:26:23.318348 21577 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 06:26:23.319447 21577 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 06:26:23.319481 21577 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 06:26:23.319494 21577 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 06:26:23.319505 21577 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 06:26:23.319515 21577 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 06:26:23.320926 21577 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.148:36221
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785997581049705-21554-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785997581049705-21554-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.12.190:34787
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.190.keytab
--principal=kudu/127.21.12.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.12.190:34787
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785997581049705-21554-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.12.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21577
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785997581049705-21554-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 05:11:02 UTC on ccycloud.quasar-hvzbbz.root.comops.site
build id 4569617
I20260806 06:26:23.321151 21577 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 06:26:23.321377 21577 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 06:26:23.326561 21585 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:26:23.326624 21589 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:26:23.326573 21584 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:26:23.326623 21577 server_base.cc:1034] running on GCE node
I20260806 06:26:23.327008 21577 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 06:26:23.327301 21577 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 06:26:23.328445 21577 hybrid_clock.cc:648] HybridClock initialized: now 1785997583328420 us; error 40 us; skew 500 ppm
Aug 06 06:26:23 dist-test-slave-4tdj krb5kdc[21563](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997583, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 06:26:23.331816 21577 init.cc:377] Logged in from keytab as kudu/127.21.12.190@KRBTEST.COM (short username kudu)
I20260806 06:26:23.332329 21577 webserver.cc:460] Webserver started at http://127.21.12.190:46121/ using document root <none> and password file <none>
I20260806 06:26:23.332584 21577 fs_manager.cc:359] Metadata directory not provided
I20260806 06:26:23.332641 21577 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 06:26:23.332767 21577 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 06:26:23.334164 21577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785997581049705-21554-0/minicluster-data/master-0/data/instance:
uuid: "2d404389bf9847e6abc80cb77f8b424f"
format_stamp: "Formatted at 2026-08-06 06:26:23 on dist-test-slave-4tdj"
server_key: "0f592daa66b4a2e9904a5b208abf72e6"
server_key_iv: "bdda431ebb365854d95aeb4f9abccc38"
server_key_version: "encryptionkey@0"
I20260806 06:26:23.334563 21577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785997581049705-21554-0/minicluster-data/master-0/wal/instance:
uuid: "2d404389bf9847e6abc80cb77f8b424f"
format_stamp: "Formatted at 2026-08-06 06:26:23 on dist-test-slave-4tdj"
server_key: "0f592daa66b4a2e9904a5b208abf72e6"
server_key_iv: "bdda431ebb365854d95aeb4f9abccc38"
server_key_version: "encryptionkey@0"
I20260806 06:26:23.335958 21577 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260806 06:26:23.336803 21604 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 06:26:23.337002 21577 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 06:26:23.337134 21577 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785997581049705-21554-0/minicluster-data/master-0/data,/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785997581049705-21554-0/minicluster-data/master-0/wal
uuid: "2d404389bf9847e6abc80cb77f8b424f"
format_stamp: "Formatted at 2026-08-06 06:26:23 on dist-test-slave-4tdj"
server_key: "0f592daa66b4a2e9904a5b208abf72e6"
server_key_iv: "bdda431ebb365854d95aeb4f9abccc38"
server_key_version: "encryptionkey@0"
I20260806 06:26:23.337208 21577 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785997581049705-21554-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785997581049705-21554-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785997581049705-21554-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 06:26:23.346969 21577 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 06:26:23.350397 21577 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 06:26:23.350641 21577 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 06:26:23.363976 21577 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.190:34787
I20260806 06:26:23.364043 21669 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.190:34787 every 8 connection(s)
I20260806 06:26:23.364681 21577 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785997581049705-21554-0/minicluster-data/master-0/data/info.pb
I20260806 06:26:23.365828 21670 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 06:26:23.368454 21670 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d404389bf9847e6abc80cb77f8b424f: Bootstrap starting.
I20260806 06:26:23.369079 21670 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2d404389bf9847e6abc80cb77f8b424f: Neither blocks nor log segments found. Creating new log.
I20260806 06:26:23.369378 21670 log.cc:824] T 00000000000000000000000000000000 P 2d404389bf9847e6abc80cb77f8b424f: Log is configured to *not* fsync() on all Append() calls
I20260806 06:26:23.370396 21670 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d404389bf9847e6abc80cb77f8b424f: No bootstrap required, opened a new log
I20260806 06:26:23.374191 21554 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHhLhT/build/release/bin/kudu as pid 21577
I20260806 06:26:23.374339 21554 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785997581049705-21554-0/minicluster-data/master-0/wal/instance
I20260806 06:26:23.374938 21670 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2d404389bf9847e6abc80cb77f8b424f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d404389bf9847e6abc80cb77f8b424f" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 34787 } }
I20260806 06:26:23.375149 21670 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2d404389bf9847e6abc80cb77f8b424f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 06:26:23.375200 21670 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2d404389bf9847e6abc80cb77f8b424f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d404389bf9847e6abc80cb77f8b424f, State: Initialized, Role: FOLLOWER
I20260806 06:26:23.375203 21554 external_mini_cluster.cc:1428] Setting key 257307804c9e88c3ba60710aa09558cc
I20260806 06:26:23.375398 21670 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2d404389bf9847e6abc80cb77f8b424f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d404389bf9847e6abc80cb77f8b424f" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 34787 } }
I20260806 06:26:23.375478 21670 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2d404389bf9847e6abc80cb77f8b424f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 06:26:23.375528 21670 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2d404389bf9847e6abc80cb77f8b424f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 06:26:23.375573 21670 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2d404389bf9847e6abc80cb77f8b424f [term 0 FOLLOWER]: Advancing to term 1
I20260806 06:26:23.376394 21670 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2d404389bf9847e6abc80cb77f8b424f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d404389bf9847e6abc80cb77f8b424f" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 34787 } }
I20260806 06:26:23.376514 21670 leader_election.cc:302] T 00000000000000000000000000000000 P 2d404389bf9847e6abc80cb77f8b424f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2d404389bf9847e6abc80cb77f8b424f; no voters:
I20260806 06:26:23.376670 21670 leader_election.cc:288] T 00000000000000000000000000000000 P 2d404389bf9847e6abc80cb77f8b424f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 06:26:23.376945 21670 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2d404389bf9847e6abc80cb77f8b424f [sys.catalog]: configured and running, proceeding with master startup.
I20260806 06:26:23.377219 21677 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2d404389bf9847e6abc80cb77f8b424f [term 1 FOLLOWER]: Leader election won for term 1
I20260806 06:26:23.377411 21677 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2d404389bf9847e6abc80cb77f8b424f [term 1 LEADER]: Becoming Leader. State: Replica: 2d404389bf9847e6abc80cb77f8b424f, State: Running, Role: LEADER
I20260806 06:26:23.377568 21677 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2d404389bf9847e6abc80cb77f8b424f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d404389bf9847e6abc80cb77f8b424f" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 34787 } }
I20260806 06:26:23.377965 21677 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d404389bf9847e6abc80cb77f8b424f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2d404389bf9847e6abc80cb77f8b424f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d404389bf9847e6abc80cb77f8b424f" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 34787 } } }
I20260806 06:26:23.378055 21676 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d404389bf9847e6abc80cb77f8b424f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2d404389bf9847e6abc80cb77f8b424f. Latest consensus state: current_term: 1 leader_uuid: "2d404389bf9847e6abc80cb77f8b424f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d404389bf9847e6abc80cb77f8b424f" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 34787 } } }
I20260806 06:26:23.378163 21676 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d404389bf9847e6abc80cb77f8b424f [sys.catalog]: This master's current role is: LEADER
I20260806 06:26:23.378122 21677 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d404389bf9847e6abc80cb77f8b424f [sys.catalog]: This master's current role is: LEADER
I20260806 06:26:23.379148 21696 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 06:26:23.379638 21696 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 06:26:23.381251 21696 catalog_manager.cc:1347] Generated new cluster ID: b529d90bbb344b2484949e0132c25d0f
I20260806 06:26:23.381302 21696 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 06 06:26:23 dist-test-slave-4tdj krb5kdc[21563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997583, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.12.190@KRBTEST.COM
I20260806 06:26:23.396282 21678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:23.376756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45706 (local address 127.21.12.190:34787)
0806 06:26:23.377128 (+ 372us) server_negotiation.cc:206] Beginning negotiation
0806 06:26:23.377137 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 06:26:23.377170 (+ 33us) server_negotiation.cc:407] Connection header received
0806 06:26:23.379577 (+ 2407us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:26:23.379584 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:26:23.379643 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:26:23.379784 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:26:23.384508 (+ 4724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:23.385884 (+ 1376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:23.387476 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:23.388103 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:23.388195 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:23.392412 (+ 4217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:26:23.392417 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:26:23.392417 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 06:26:23.392430 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:26:23.394353 (+ 1923us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:26:23.394796 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:26:23.394799 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:26:23.394799 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:26:23.394902 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:26:23.395173 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:26:23.395174 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:26:23.395175 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:26:23.395443 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:26:23.395543 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0806 06:26:23.395903 (+ 360us) server_negotiation.cc:299] Negotiation successful
0806 06:26:23.396055 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":120,"threads_started":1}
I20260806 06:26:23.415171 21696 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 06:26:23.415844 21696 catalog_manager.cc:1504] Loading token signing keys...
I20260806 06:26:23.432235 21696 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2d404389bf9847e6abc80cb77f8b424f: Generated new TSK 0
I20260806 06:26:23.432464 21696 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.12.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.12.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.12.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.129.keytab.
Entry for principal kudu/127.21.12.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.129.keytab.
I20260806 06:26:23.463274 21554 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHhLhT/build/release/bin/kudu
/tmp/dist-test-taskFHhLhT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785997581049705-21554-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785997581049705-21554-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785997581049705-21554-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785997581049705-21554-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.12.129:0
--local_ip_for_outbound_sockets=127.21.12.129
--webserver_interface=127.21.12.129
--webserver_port=0
--tserver_master_addrs=127.21.12.190:34787
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.129.keytab
--principal=kudu/127.21.12.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.148:36221
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785997581049705-21554-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785997581049705-21554-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785997581049705-21554-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 06:26:23.546373 21707 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 06:26:23.546586 21707 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 06:26:23.546617 21707 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 06:26:23.547732 21707 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 06:26:23.547771 21707 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 06:26:23.547806 21707 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.12.129
I20260806 06:26:23.549244 21707 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.148:36221
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785997581049705-21554-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785997581049705-21554-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.129.keytab
--principal=kudu/127.21.12.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.12.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785997581049705-21554-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.12.129
--webserver_port=0
--tserver_master_addrs=127.21.12.190:34787
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21707
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.12.129
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785997581049705-21554-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 05:11:02 UTC on ccycloud.quasar-hvzbbz.root.comops.site
build id 4569617
I20260806 06:26:23.549474 21707 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 06:26:23.549678 21707 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 06:26:23.554528 21717 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:26:23.554548 21715 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:26:23.554564 21713 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:26:23.554656 21707 server_base.cc:1034] running on GCE node
I20260806 06:26:23.554898 21707 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 06:26:23.555112 21707 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 06:26:23.556257 21707 hybrid_clock.cc:648] HybridClock initialized: now 1785997583556247 us; error 34 us; skew 500 ppm
Aug 06 06:26:23 dist-test-slave-4tdj krb5kdc[21563](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997583, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 06:26:23.558732 21707 init.cc:377] Logged in from keytab as kudu/127.21.12.129@KRBTEST.COM (short username kudu)
I20260806 06:26:23.559103 21707 webserver.cc:460] Webserver started at http://127.21.12.129:40713/ using document root <none> and password file <none>
I20260806 06:26:23.559276 21707 fs_manager.cc:359] Metadata directory not provided
I20260806 06:26:23.559314 21707 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 06:26:23.559403 21707 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 06:26:23.560359 21707 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785997581049705-21554-0/minicluster-data/ts-0/data/instance:
uuid: "c3ff43db962042ac8fbfc736dc4f2696"
format_stamp: "Formatted at 2026-08-06 06:26:23 on dist-test-slave-4tdj"
server_key: "d35c52f1648e474ab684e49c5ac494c0"
server_key_iv: "02242b4b6509ef8c8feb779d0083d572"
server_key_version: "encryptionkey@0"
I20260806 06:26:23.560657 21707 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785997581049705-21554-0/minicluster-data/ts-0/wal/instance:
uuid: "c3ff43db962042ac8fbfc736dc4f2696"
format_stamp: "Formatted at 2026-08-06 06:26:23 on dist-test-slave-4tdj"
server_key: "d35c52f1648e474ab684e49c5ac494c0"
server_key_iv: "02242b4b6509ef8c8feb779d0083d572"
server_key_version: "encryptionkey@0"
I20260806 06:26:23.561810 21707 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260806 06:26:23.562466 21724 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 06:26:23.562628 21707 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260806 06:26:23.562705 21707 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785997581049705-21554-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785997581049705-21554-0/minicluster-data/ts-0/wal
uuid: "c3ff43db962042ac8fbfc736dc4f2696"
format_stamp: "Formatted at 2026-08-06 06:26:23 on dist-test-slave-4tdj"
server_key: "d35c52f1648e474ab684e49c5ac494c0"
server_key_iv: "02242b4b6509ef8c8feb779d0083d572"
server_key_version: "encryptionkey@0"
I20260806 06:26:23.562773 21707 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785997581049705-21554-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785997581049705-21554-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785997581049705-21554-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 06:26:23.572494 21707 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 06:26:23.574419 21707 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 06:26:23.574550 21707 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 06:26:23.574748 21707 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 06:26:23.575042 21707 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 06:26:23.575075 21707 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 06:26:23.575114 21707 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 06:26:23.575137 21707 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 06:26:23.581403 21707 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.129:38501
I20260806 06:26:23.581460 21876 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.129:38501 every 8 connection(s)
I20260806 06:26:23.581754 21707 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785997581049705-21554-0/minicluster-data/ts-0/data/info.pb
Aug 06 06:26:23 dist-test-slave-4tdj krb5kdc[21563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997583, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.129@KRBTEST.COM for kudu/127.21.12.190@KRBTEST.COM
I20260806 06:26:23.588485 21554 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHhLhT/build/release/bin/kudu as pid 21707
I20260806 06:26:23.588559 21554 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785997581049705-21554-0/minicluster-data/ts-0/wal/instance
I20260806 06:26:23.588699 21554 external_mini_cluster.cc:1428] Setting key f97678db4ea46d609caeceb670eebeea
I20260806 06:26:23.590785 21678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:23.582469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.129:53689 (local address 127.21.12.190:34787)
0806 06:26:23.582526 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0806 06:26:23.582527 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0806 06:26:23.582749 (+ 222us) server_negotiation.cc:407] Connection header received
0806 06:26:23.583220 (+ 471us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:26:23.583221 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:26:23.583236 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:26:23.583274 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:26:23.584280 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:23.584783 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:23.585694 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:23.585890 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:23.585944 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:23.587989 (+ 2045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:26:23.587991 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:26:23.587991 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 06:26:23.587999 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:26:23.589481 (+ 1482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:26:23.589902 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:26:23.589903 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:26:23.589903 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:26:23.589956 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:26:23.590182 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:26:23.590182 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:26:23.590182 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:26:23.590274 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:26:23.590378 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0806 06:26:23.590655 (+ 277us) server_negotiation.cc:299] Negotiation successful
0806 06:26:23.590721 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260806 06:26:23.590998 21879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:23.582469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:34787 (local address 127.21.12.129:53689)
0806 06:26:23.582619 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0806 06:26:23.582654 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0806 06:26:23.583125 (+ 471us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:26:23.583328 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:26:23.583331 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:26:23.584001 (+ 670us) client_negotiation.cc:189] Negotiated authn=SASL
0806 06:26:23.584193 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:23.584199 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:23.584849 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:23.584850 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:23.585616 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:23.585618 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:23.585949 (+ 331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:23.585950 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:23.586040 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:23.586496 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 06:26:23.586504 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0806 06:26:23.587900 (+ 1396us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 06:26:23.589557 (+ 1657us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:26:23.589565 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:26:23.589567 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:26:23.589836 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:26:23.590007 (+ 171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:26:23.590008 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:26:23.590008 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:26:23.590130 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:26:23.590372 (+ 242us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 06:26:23.590379 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 06:26:23.590527 (+ 148us) client_negotiation.cc:769] Sending connection context
0806 06:26:23.590630 (+ 103us) client_negotiation.cc:240] Negotiation successful
0806 06:26:23.590788 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":87,"threads_started":1}
I20260806 06:26:23.591462 21877 heartbeater.cc:347] Connected to a master server at 127.21.12.190:34787
I20260806 06:26:23.591557 21877 heartbeater.cc:464] Registering TS with master...
I20260806 06:26:23.591740 21877 heartbeater.cc:511] Master 127.21.12.190:34787 requested a full tablet report, sending...
I20260806 06:26:23.592214 21624 ts_manager.cc:194] Registered new tserver with Master: c3ff43db962042ac8fbfc736dc4f2696 (127.21.12.129:38501)
I20260806 06:26:23.592805 21624 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.12.129@KRBTEST.COM'} at 127.21.12.129:53689
WARNING: no policy specified for kudu/127.21.12.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.12.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.12.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.130.keytab.
Entry for principal kudu/127.21.12.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.130.keytab.
I20260806 06:26:23.613550 21554 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHhLhT/build/release/bin/kudu
/tmp/dist-test-taskFHhLhT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785997581049705-21554-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785997581049705-21554-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785997581049705-21554-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785997581049705-21554-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.12.130:0
--local_ip_for_outbound_sockets=127.21.12.130
--webserver_interface=127.21.12.130
--webserver_port=0
--tserver_master_addrs=127.21.12.190:34787
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.130.keytab
--principal=kudu/127.21.12.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.148:36221
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785997581049705-21554-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785997581049705-21554-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785997581049705-21554-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 06:26:23.717697 21884 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 06:26:23.717908 21884 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 06:26:23.717940 21884 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 06:26:23.719156 21884 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 06:26:23.719198 21884 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 06:26:23.719234 21884 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.12.130
I20260806 06:26:23.720889 21884 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.148:36221
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785997581049705-21554-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785997581049705-21554-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.130.keytab
--principal=kudu/127.21.12.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.12.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785997581049705-21554-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.12.130
--webserver_port=0
--tserver_master_addrs=127.21.12.190:34787
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21884
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.12.130
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785997581049705-21554-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 05:11:02 UTC on ccycloud.quasar-hvzbbz.root.comops.site
build id 4569617
I20260806 06:26:23.721138 21884 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 06:26:23.721347 21884 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 06:26:23.726619 21895 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:26:23.726737 21892 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:26:23.726779 21884 server_base.cc:1034] running on GCE node
W20260806 06:26:23.726863 21890 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:26:23.727059 21884 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 06:26:23.727247 21884 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 06:26:23.728377 21884 hybrid_clock.cc:648] HybridClock initialized: now 1785997583728364 us; error 29 us; skew 500 ppm
Aug 06 06:26:23 dist-test-slave-4tdj krb5kdc[21563](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997583, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 06:26:23.731388 21884 init.cc:377] Logged in from keytab as kudu/127.21.12.130@KRBTEST.COM (short username kudu)
I20260806 06:26:23.731796 21884 webserver.cc:460] Webserver started at http://127.21.12.130:42113/ using document root <none> and password file <none>
I20260806 06:26:23.731999 21884 fs_manager.cc:359] Metadata directory not provided
I20260806 06:26:23.732041 21884 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 06:26:23.732161 21884 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 06:26:23.733312 21884 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785997581049705-21554-0/minicluster-data/ts-1/data/instance:
uuid: "c2b60461be8945fd85c206349ebf3ee9"
format_stamp: "Formatted at 2026-08-06 06:26:23 on dist-test-slave-4tdj"
server_key: "8603dd5b50990e0b187707fa4fe1ed2b"
server_key_iv: "f9f473789e51a9d5992c6a9daefbb127"
server_key_version: "encryptionkey@0"
I20260806 06:26:23.733700 21884 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785997581049705-21554-0/minicluster-data/ts-1/wal/instance:
uuid: "c2b60461be8945fd85c206349ebf3ee9"
format_stamp: "Formatted at 2026-08-06 06:26:23 on dist-test-slave-4tdj"
server_key: "8603dd5b50990e0b187707fa4fe1ed2b"
server_key_iv: "f9f473789e51a9d5992c6a9daefbb127"
server_key_version: "encryptionkey@0"
I20260806 06:26:23.735080 21884 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260806 06:26:23.735862 21904 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 06:26:23.736007 21884 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260806 06:26:23.736088 21884 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785997581049705-21554-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785997581049705-21554-0/minicluster-data/ts-1/wal
uuid: "c2b60461be8945fd85c206349ebf3ee9"
format_stamp: "Formatted at 2026-08-06 06:26:23 on dist-test-slave-4tdj"
server_key: "8603dd5b50990e0b187707fa4fe1ed2b"
server_key_iv: "f9f473789e51a9d5992c6a9daefbb127"
server_key_version: "encryptionkey@0"
I20260806 06:26:23.736153 21884 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785997581049705-21554-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785997581049705-21554-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785997581049705-21554-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 06:26:23.749637 21884 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 06:26:23.751945 21884 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 06:26:23.752123 21884 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 06:26:23.752458 21884 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 06:26:23.752801 21884 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 06:26:23.752837 21884 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 06:26:23.752866 21884 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 06:26:23.752882 21884 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 06:26:23.767894 21884 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.130:39109
I20260806 06:26:23.767931 22053 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.130:39109 every 8 connection(s)
I20260806 06:26:23.768278 21884 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785997581049705-21554-0/minicluster-data/ts-1/data/info.pb
I20260806 06:26:23.768488 21554 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHhLhT/build/release/bin/kudu as pid 21884
I20260806 06:26:23.768567 21554 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785997581049705-21554-0/minicluster-data/ts-1/wal/instance
I20260806 06:26:23.768733 21554 external_mini_cluster.cc:1428] Setting key ac29f7717ab32421325d2dd065cbc701
Aug 06 06:26:23 dist-test-slave-4tdj krb5kdc[21563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997583, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.130@KRBTEST.COM for kudu/127.21.12.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.21.12.131@KRBTEST.COM; defaulting to no policy
I20260806 06:26:23.779429 21678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:23.769117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.130:35165 (local address 127.21.12.190:34787)
0806 06:26:23.769165 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0806 06:26:23.769167 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:26:23.769609 (+ 442us) server_negotiation.cc:407] Connection header received
0806 06:26:23.770271 (+ 662us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:26:23.770271 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:26:23.770284 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:26:23.770376 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:26:23.771706 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:23.772295 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:23.773732 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:23.773922 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:23.773978 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:23.776594 (+ 2616us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:26:23.776596 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:26:23.776596 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 06:26:23.776604 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:26:23.777955 (+ 1351us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:26:23.778498 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:26:23.778499 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:26:23.778499 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:26:23.778573 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:26:23.778836 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:26:23.778836 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:26:23.778836 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:26:23.778931 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:26:23.778976 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0806 06:26:23.779296 (+ 320us) server_negotiation.cc:299] Negotiation successful
0806 06:26:23.779364 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260806 06:26:23.779620 22059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:23.769122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:34787 (local address 127.21.12.130:35165)
0806 06:26:23.769458 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0806 06:26:23.769505 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0806 06:26:23.770166 (+ 661us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:26:23.770439 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:26:23.770443 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:26:23.771194 (+ 751us) client_negotiation.cc:189] Negotiated authn=SASL
0806 06:26:23.771610 (+ 416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:23.771617 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:23.772397 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:23.772398 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:23.773649 (+ 1251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:23.773652 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:23.774010 (+ 358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:23.774011 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:23.774213 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:23.774792 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 06:26:23.774803 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0806 06:26:23.776480 (+ 1677us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 06:26:23.778048 (+ 1568us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:26:23.778055 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:26:23.778058 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:26:23.778426 (+ 368us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:26:23.778637 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:26:23.778638 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:26:23.778638 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:26:23.778777 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:26:23.778982 (+ 205us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 06:26:23.778987 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 06:26:23.779157 (+ 170us) client_negotiation.cc:769] Sending connection context
0806 06:26:23.779278 (+ 121us) client_negotiation.cc:240] Negotiation successful
0806 06:26:23.779426 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":176,"threads_started":1}
I20260806 06:26:23.780069 22054 heartbeater.cc:347] Connected to a master server at 127.21.12.190:34787
I20260806 06:26:23.780148 22054 heartbeater.cc:464] Registering TS with master...
I20260806 06:26:23.780318 22054 heartbeater.cc:511] Master 127.21.12.190:34787 requested a full tablet report, sending...
I20260806 06:26:23.780659 21624 ts_manager.cc:194] Registered new tserver with Master: c2b60461be8945fd85c206349ebf3ee9 (127.21.12.130:39109)
I20260806 06:26:23.780984 21624 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.12.130@KRBTEST.COM'} at 127.21.12.130:35165
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.12.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.12.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.131.keytab.
Entry for principal kudu/127.21.12.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.131.keytab.
I20260806 06:26:23.796868 21554 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHhLhT/build/release/bin/kudu
/tmp/dist-test-taskFHhLhT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785997581049705-21554-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785997581049705-21554-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785997581049705-21554-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785997581049705-21554-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.12.131:0
--local_ip_for_outbound_sockets=127.21.12.131
--webserver_interface=127.21.12.131
--webserver_port=0
--tserver_master_addrs=127.21.12.190:34787
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.131.keytab
--principal=kudu/127.21.12.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.148:36221
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785997581049705-21554-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785997581049705-21554-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785997581049705-21554-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 06:26:23.900225 22063 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 06:26:23.900437 22063 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 06:26:23.900460 22063 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 06:26:23.901710 22063 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 06:26:23.901774 22063 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 06:26:23.901805 22063 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.12.131
I20260806 06:26:23.903725 22063 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.148:36221
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785997581049705-21554-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785997581049705-21554-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.131.keytab
--principal=kudu/127.21.12.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.12.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785997581049705-21554-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.12.131
--webserver_port=0
--tserver_master_addrs=127.21.12.190:34787
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22063
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.12.131
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785997581049705-21554-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 05:11:02 UTC on ccycloud.quasar-hvzbbz.root.comops.site
build id 4569617
I20260806 06:26:23.903952 22063 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 06:26:23.904174 22063 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 06:26:23.910132 22069 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:26:23.910100 22071 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:26:23.910171 22063 server_base.cc:1034] running on GCE node
W20260806 06:26:23.910117 22073 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:26:23.910527 22063 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 06:26:23.910751 22063 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 06:26:23.911887 22063 hybrid_clock.cc:648] HybridClock initialized: now 1785997583911869 us; error 28 us; skew 500 ppm
Aug 06 06:26:23 dist-test-slave-4tdj krb5kdc[21563](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997583, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 06:26:23.914260 22063 init.cc:377] Logged in from keytab as kudu/127.21.12.131@KRBTEST.COM (short username kudu)
I20260806 06:26:23.914700 22063 webserver.cc:460] Webserver started at http://127.21.12.131:41295/ using document root <none> and password file <none>
I20260806 06:26:23.914888 22063 fs_manager.cc:359] Metadata directory not provided
I20260806 06:26:23.914933 22063 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 06:26:23.915055 22063 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 06:26:23.916051 22063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785997581049705-21554-0/minicluster-data/ts-2/data/instance:
uuid: "0ca0d1c36d2345618292097a2ddc64b5"
format_stamp: "Formatted at 2026-08-06 06:26:23 on dist-test-slave-4tdj"
server_key: "3396492f50624a69b4c1dc3ce89fe0ae"
server_key_iv: "7b9e191f2ff879cf388fafe1f5174f7b"
server_key_version: "encryptionkey@0"
I20260806 06:26:23.916375 22063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785997581049705-21554-0/minicluster-data/ts-2/wal/instance:
uuid: "0ca0d1c36d2345618292097a2ddc64b5"
format_stamp: "Formatted at 2026-08-06 06:26:23 on dist-test-slave-4tdj"
server_key: "3396492f50624a69b4c1dc3ce89fe0ae"
server_key_iv: "7b9e191f2ff879cf388fafe1f5174f7b"
server_key_version: "encryptionkey@0"
I20260806 06:26:23.917704 22063 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260806 06:26:23.918500 22081 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 06:26:23.918692 22063 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260806 06:26:23.918785 22063 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785997581049705-21554-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785997581049705-21554-0/minicluster-data/ts-2/wal
uuid: "0ca0d1c36d2345618292097a2ddc64b5"
format_stamp: "Formatted at 2026-08-06 06:26:23 on dist-test-slave-4tdj"
server_key: "3396492f50624a69b4c1dc3ce89fe0ae"
server_key_iv: "7b9e191f2ff879cf388fafe1f5174f7b"
server_key_version: "encryptionkey@0"
I20260806 06:26:23.918843 22063 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785997581049705-21554-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785997581049705-21554-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785997581049705-21554-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 06:26:23.928674 22063 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 06:26:23.930934 22063 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 06:26:23.931105 22063 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 06:26:23.931430 22063 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 06:26:23.931797 22063 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 06:26:23.931846 22063 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 06:26:23.931876 22063 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 06:26:23.931900 22063 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 06:26:23.943010 22063 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.131:41583
I20260806 06:26:23.943064 22228 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.131:41583 every 8 connection(s)
I20260806 06:26:23.943398 22063 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785997581049705-21554-0/minicluster-data/ts-2/data/info.pb
Aug 06 06:26:23 dist-test-slave-4tdj krb5kdc[21563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997583, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.131@KRBTEST.COM for kudu/127.21.12.190@KRBTEST.COM
I20260806 06:26:23.952162 21678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:23.944160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.131:37139 (local address 127.21.12.190:34787)
0806 06:26:23.944205 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0806 06:26:23.944207 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:26:23.944497 (+ 290us) server_negotiation.cc:407] Connection header received
0806 06:26:23.944971 (+ 474us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:26:23.944972 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:26:23.944985 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:26:23.945037 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:26:23.945966 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:23.946509 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:23.947458 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:23.947651 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:23.947707 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:23.949715 (+ 2008us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:26:23.949716 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:26:23.949716 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 06:26:23.949724 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:26:23.950947 (+ 1223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:26:23.951344 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:26:23.951345 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:26:23.951345 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:26:23.951416 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:26:23.951628 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:26:23.951629 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:26:23.951629 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:26:23.951750 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:26:23.951802 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0806 06:26:23.951995 (+ 193us) server_negotiation.cc:299] Negotiation successful
0806 06:26:23.952072 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260806 06:26:23.952171 21554 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHhLhT/build/release/bin/kudu as pid 22063
I20260806 06:26:23.952261 21554 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785997581049705-21554-0/minicluster-data/ts-2/wal/instance
I20260806 06:26:23.952309 22231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:23.944150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:34787 (local address 127.21.12.131:37139)
0806 06:26:23.944382 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0806 06:26:23.944417 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0806 06:26:23.944877 (+ 460us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:26:23.945124 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:26:23.945129 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:26:23.945687 (+ 558us) client_negotiation.cc:189] Negotiated authn=SASL
0806 06:26:23.945870 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:23.945875 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:23.946604 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:23.946606 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:23.947376 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:23.947378 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:23.947720 (+ 342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:23.947721 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:23.947832 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:23.948264 (+ 432us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 06:26:23.948273 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0806 06:26:23.949632 (+ 1359us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 06:26:23.951038 (+ 1406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:26:23.951044 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:26:23.951046 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:26:23.951271 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:26:23.951485 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:26:23.951485 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:26:23.951486 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:26:23.951581 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:26:23.951797 (+ 216us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 06:26:23.951800 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 06:26:23.951903 (+ 103us) client_negotiation.cc:769] Sending connection context
0806 06:26:23.951974 (+ 71us) client_negotiation.cc:240] Negotiation successful
0806 06:26:23.952094 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":159,"threads_started":1}
I20260806 06:26:23.952418 21554 external_mini_cluster.cc:1428] Setting key 19bc63057a4860439eebf616c2b5ca84
I20260806 06:26:23.952842 22229 heartbeater.cc:347] Connected to a master server at 127.21.12.190:34787
I20260806 06:26:23.952939 22229 heartbeater.cc:464] Registering TS with master...
I20260806 06:26:23.953115 22229 heartbeater.cc:511] Master 127.21.12.190:34787 requested a full tablet report, sending...
I20260806 06:26:23.953445 21624 ts_manager.cc:194] Registered new tserver with Master: 0ca0d1c36d2345618292097a2ddc64b5 (127.21.12.131:41583)
I20260806 06:26:23.953805 21624 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.12.131@KRBTEST.COM'} at 127.21.12.131:37139
I20260806 06:26:23.964318 21554 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 06 06:26:23 dist-test-slave-4tdj krb5kdc[21563](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997583, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 06 06:26:23 dist-test-slave-4tdj krb5kdc[21563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997583, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.21.12.129@KRBTEST.COM
I20260806 06:26:23.988493 22241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:23.981065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45858 (local address 127.21.12.129:38501)
0806 06:26:23.981341 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0806 06:26:23.981345 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 06:26:23.981357 (+ 12us) server_negotiation.cc:407] Connection header received
0806 06:26:23.981384 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:26:23.981386 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:26:23.981406 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:26:23.981487 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:26:23.982023 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:23.982840 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:23.983547 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:23.983955 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:23.984013 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:23.986108 (+ 2095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:26:23.986117 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:26:23.986117 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 06:26:23.986124 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:26:23.987573 (+ 1449us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:26:23.987862 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:26:23.987865 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:26:23.987865 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:26:23.987919 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:26:23.988054 (+ 135us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:26:23.988054 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:26:23.988054 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:26:23.988171 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:26:23.988265 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0806 06:26:23.988320 (+ 55us) server_negotiation.cc:299] Negotiation successful
0806 06:26:23.988382 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":230,"thread_start_us":140,"threads_started":1}
W20260806 06:26:23.988867 21763 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45858
Aug 06 06:26:23 dist-test-slave-4tdj krb5kdc[21563](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997583, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 06 06:26:24 dist-test-slave-4tdj krb5kdc[21563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997583, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.12.129@KRBTEST.COM
I20260806 06:26:24.010604 22241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:24.004160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45870 (local address 127.21.12.129:38501)
0806 06:26:24.004208 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0806 06:26:24.004211 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 06:26:24.004290 (+ 79us) server_negotiation.cc:407] Connection header received
0806 06:26:24.004336 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:26:24.004337 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:26:24.004347 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:26:24.004392 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:26:24.004888 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:24.005371 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:24.006089 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:24.006286 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:24.006359 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:24.008446 (+ 2087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:26:24.008447 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:26:24.008448 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 06:26:24.008454 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:26:24.009652 (+ 1198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:26:24.009937 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:26:24.009938 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:26:24.009938 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:26:24.009982 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:26:24.010111 (+ 129us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:26:24.010112 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:26:24.010112 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:26:24.010193 (+ 81us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:26:24.010275 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0806 06:26:24.010461 (+ 186us) server_negotiation.cc:299] Negotiation successful
0806 06:26:24.010529 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260806 06:26:24.011121 21554 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHhLhT/build/release/bin/kudu with pid 21707
I20260806 06:26:24.020982 21554 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHhLhT/build/release/bin/kudu with pid 21884
I20260806 06:26:24.034682 21554 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHhLhT/build/release/bin/kudu with pid 22063
I20260806 06:26:24.044013 21554 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHhLhT/build/release/bin/kudu with pid 21577
2026-08-06T06:26:24Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (3005 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785997581049705-21554-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 06:26:24 dist-test-slave-4tdj krb5kdc[22249](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 06:26:24 dist-test-slave-4tdj krb5kdc[22249](info): set up 2 sockets
Aug 06 06:26:24 dist-test-slave-4tdj krb5kdc[22249](info): commencing operation
krb5kdc: starting...
W20260806 06:26:25.164212 21554 mini_kdc.cc:121] Time spent starting KDC: real 1.096s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785997581049705-21554-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785997581049705-21554-0/krb5kdc/test-user.keytab.
Aug 06 06:26:25 dist-test-slave-4tdj krb5kdc[22249](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997585, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T06:26:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T06:26:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.12.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.12.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.12.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.190.keytab.
Entry for principal kudu/127.21.12.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.190.keytab.
I20260806 06:26:25.276604 21554 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHhLhT/build/release/bin/kudu
/tmp/dist-test-taskFHhLhT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785997581049705-21554-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785997581049705-21554-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785997581049705-21554-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785997581049705-21554-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.12.190:41629
--webserver_interface=127.21.12.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.190.keytab
--principal=kudu/127.21.12.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.148:45247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.12.190:41629
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785997581049705-21554-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785997581049705-21554-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785997581049705-21554-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 06:26:25.357983 22267 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 06:26:25.358230 22267 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 06:26:25.358256 22267 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 06:26:25.359452 22267 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 06:26:25.359499 22267 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 06:26:25.359512 22267 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 06:26:25.359524 22267 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 06:26:25.359534 22267 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 06:26:25.361063 22267 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.148:45247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785997581049705-21554-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785997581049705-21554-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.12.190:41629
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.190.keytab
--principal=kudu/127.21.12.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.12.190:41629
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785997581049705-21554-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.12.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22267
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785997581049705-21554-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 05:11:02 UTC on ccycloud.quasar-hvzbbz.root.comops.site
build id 4569617
I20260806 06:26:25.361296 22267 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 06:26:25.361497 22267 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 06:26:25.366335 22276 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:26:25.366366 22272 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:26:25.366451 22267 server_base.cc:1034] running on GCE node
W20260806 06:26:25.366344 22273 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:26:25.366720 22267 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 06:26:25.367034 22267 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 06:26:25.368213 22267 hybrid_clock.cc:648] HybridClock initialized: now 1785997585368196 us; error 28 us; skew 500 ppm
Aug 06 06:26:25 dist-test-slave-4tdj krb5kdc[22249](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997585, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 06:26:25.370584 22267 init.cc:377] Logged in from keytab as kudu/127.21.12.190@KRBTEST.COM (short username kudu)
I20260806 06:26:25.371002 22267 webserver.cc:460] Webserver started at http://127.21.12.190:45975/ using document root <none> and password file <none>
I20260806 06:26:25.371258 22267 fs_manager.cc:359] Metadata directory not provided
I20260806 06:26:25.371316 22267 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 06:26:25.371443 22267 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 06:26:25.372486 22267 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785997581049705-21554-0/minicluster-data/master-0/data/instance:
uuid: "1bfff40ce62f4c4db8505a088758cf76"
format_stamp: "Formatted at 2026-08-06 06:26:25 on dist-test-slave-4tdj"
server_key: "661a924a556b1af629e3b24b3debabc7"
server_key_iv: "6a06ed67a1548990317e7e908d1ebc0e"
server_key_version: "encryptionkey@0"
I20260806 06:26:25.372850 22267 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785997581049705-21554-0/minicluster-data/master-0/wal/instance:
uuid: "1bfff40ce62f4c4db8505a088758cf76"
format_stamp: "Formatted at 2026-08-06 06:26:25 on dist-test-slave-4tdj"
server_key: "661a924a556b1af629e3b24b3debabc7"
server_key_iv: "6a06ed67a1548990317e7e908d1ebc0e"
server_key_version: "encryptionkey@0"
I20260806 06:26:25.374065 22267 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260806 06:26:25.374802 22283 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 06:26:25.374948 22267 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260806 06:26:25.375032 22267 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785997581049705-21554-0/minicluster-data/master-0/data,/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785997581049705-21554-0/minicluster-data/master-0/wal
uuid: "1bfff40ce62f4c4db8505a088758cf76"
format_stamp: "Formatted at 2026-08-06 06:26:25 on dist-test-slave-4tdj"
server_key: "661a924a556b1af629e3b24b3debabc7"
server_key_iv: "6a06ed67a1548990317e7e908d1ebc0e"
server_key_version: "encryptionkey@0"
I20260806 06:26:25.375097 22267 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785997581049705-21554-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785997581049705-21554-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785997581049705-21554-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 06:26:25.386277 22267 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 06:26:25.388000 22267 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 06:26:25.388132 22267 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 06:26:25.391873 22267 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.190:41629
I20260806 06:26:25.391906 22348 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.190:41629 every 8 connection(s)
I20260806 06:26:25.392212 22267 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785997581049705-21554-0/minicluster-data/master-0/data/info.pb
I20260806 06:26:25.393036 22349 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 06:26:25.395280 22349 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1bfff40ce62f4c4db8505a088758cf76: Bootstrap starting.
I20260806 06:26:25.395804 22349 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1bfff40ce62f4c4db8505a088758cf76: Neither blocks nor log segments found. Creating new log.
I20260806 06:26:25.396040 22349 log.cc:824] T 00000000000000000000000000000000 P 1bfff40ce62f4c4db8505a088758cf76: Log is configured to *not* fsync() on all Append() calls
I20260806 06:26:25.396673 22349 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1bfff40ce62f4c4db8505a088758cf76: No bootstrap required, opened a new log
I20260806 06:26:25.398157 22349 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1bfff40ce62f4c4db8505a088758cf76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bfff40ce62f4c4db8505a088758cf76" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 41629 } }
I20260806 06:26:25.398316 22349 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1bfff40ce62f4c4db8505a088758cf76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 06:26:25.398352 22349 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1bfff40ce62f4c4db8505a088758cf76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1bfff40ce62f4c4db8505a088758cf76, State: Initialized, Role: FOLLOWER
I20260806 06:26:25.398478 22349 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1bfff40ce62f4c4db8505a088758cf76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bfff40ce62f4c4db8505a088758cf76" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 41629 } }
I20260806 06:26:25.398566 22349 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1bfff40ce62f4c4db8505a088758cf76 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 06:26:25.398590 22349 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1bfff40ce62f4c4db8505a088758cf76 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 06:26:25.398618 22349 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1bfff40ce62f4c4db8505a088758cf76 [term 0 FOLLOWER]: Advancing to term 1
I20260806 06:26:25.399080 22349 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1bfff40ce62f4c4db8505a088758cf76 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bfff40ce62f4c4db8505a088758cf76" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 41629 } }
I20260806 06:26:25.399163 22349 leader_election.cc:302] T 00000000000000000000000000000000 P 1bfff40ce62f4c4db8505a088758cf76 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1bfff40ce62f4c4db8505a088758cf76; no voters:
I20260806 06:26:25.399286 22349 leader_election.cc:288] T 00000000000000000000000000000000 P 1bfff40ce62f4c4db8505a088758cf76 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 06:26:25.399333 22355 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1bfff40ce62f4c4db8505a088758cf76 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 06:26:25.399494 22355 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1bfff40ce62f4c4db8505a088758cf76 [term 1 LEADER]: Becoming Leader. State: Replica: 1bfff40ce62f4c4db8505a088758cf76, State: Running, Role: LEADER
I20260806 06:26:25.399500 22349 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1bfff40ce62f4c4db8505a088758cf76 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 06:26:25.399590 22355 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1bfff40ce62f4c4db8505a088758cf76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bfff40ce62f4c4db8505a088758cf76" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 41629 } }
I20260806 06:26:25.399995 22358 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1bfff40ce62f4c4db8505a088758cf76 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1bfff40ce62f4c4db8505a088758cf76. Latest consensus state: current_term: 1 leader_uuid: "1bfff40ce62f4c4db8505a088758cf76" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bfff40ce62f4c4db8505a088758cf76" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 41629 } } }
I20260806 06:26:25.400096 22358 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1bfff40ce62f4c4db8505a088758cf76 [sys.catalog]: This master's current role is: LEADER
I20260806 06:26:25.400270 22357 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1bfff40ce62f4c4db8505a088758cf76 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1bfff40ce62f4c4db8505a088758cf76" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bfff40ce62f4c4db8505a088758cf76" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 41629 } } }
I20260806 06:26:25.400375 22367 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 06:26:25.400400 22357 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1bfff40ce62f4c4db8505a088758cf76 [sys.catalog]: This master's current role is: LEADER
I20260806 06:26:25.400894 22367 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 06:26:25.401432 21554 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHhLhT/build/release/bin/kudu as pid 22267
I20260806 06:26:25.401518 21554 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785997581049705-21554-0/minicluster-data/master-0/wal/instance
I20260806 06:26:25.401678 21554 external_mini_cluster.cc:1428] Setting key 4c30b8607f4130dc03c9986117c181ed
I20260806 06:26:25.402347 22367 catalog_manager.cc:1347] Generated new cluster ID: 95e2f53611b54f67aa2596268632800e
I20260806 06:26:25.402400 22367 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 06:26:25.406404 22367 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 06:26:25.406864 22367 catalog_manager.cc:1504] Loading token signing keys...
I20260806 06:26:25.410125 22367 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1bfff40ce62f4c4db8505a088758cf76: Generated new TSK 0
I20260806 06:26:25.410272 22367 catalog_manager.cc:1514] Initializing in-progress tserver states...
Aug 06 06:26:25 dist-test-slave-4tdj krb5kdc[22249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997585, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.12.190@KRBTEST.COM
I20260806 06:26:25.415963 22386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:25.402114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39650 (local address 127.21.12.190:41629)
0806 06:26:25.402251 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0806 06:26:25.402261 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 06:26:25.405930 (+ 3669us) server_negotiation.cc:407] Connection header received
0806 06:26:25.406058 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:26:25.406066 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:26:25.406122 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:26:25.406256 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:26:25.407043 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:25.408441 (+ 1398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:25.409452 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:25.410057 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:25.410160 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:25.412734 (+ 2574us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:26:25.412738 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:26:25.412738 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 06:26:25.412750 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:26:25.414525 (+ 1775us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:26:25.414850 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:26:25.414853 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:26:25.414853 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:26:25.414959 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:26:25.415116 (+ 157us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:26:25.415117 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:26:25.415117 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:26:25.415379 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:26:25.415507 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0806 06:26:25.415589 (+ 82us) server_negotiation.cc:299] Negotiation successful
0806 06:26:25.415735 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92,"thread_start_us":63,"threads_started":1}
WARNING: no policy specified for kudu/127.21.12.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.12.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.12.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.129.keytab.
Entry for principal kudu/127.21.12.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.129.keytab.
I20260806 06:26:25.439110 21554 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHhLhT/build/release/bin/kudu
/tmp/dist-test-taskFHhLhT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785997581049705-21554-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785997581049705-21554-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785997581049705-21554-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785997581049705-21554-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.12.129:0
--local_ip_for_outbound_sockets=127.21.12.129
--webserver_interface=127.21.12.129
--webserver_port=0
--tserver_master_addrs=127.21.12.190:41629
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.129.keytab
--principal=kudu/127.21.12.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.148:45247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785997581049705-21554-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785997581049705-21554-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785997581049705-21554-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 06:26:25.518725 22389 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 06:26:25.518927 22389 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 06:26:25.518960 22389 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 06:26:25.520081 22389 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 06:26:25.520120 22389 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 06:26:25.520154 22389 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.12.129
I20260806 06:26:25.521628 22389 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.148:45247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785997581049705-21554-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785997581049705-21554-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.129.keytab
--principal=kudu/127.21.12.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.12.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785997581049705-21554-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.12.129
--webserver_port=0
--tserver_master_addrs=127.21.12.190:41629
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22389
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.12.129
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785997581049705-21554-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 05:11:02 UTC on ccycloud.quasar-hvzbbz.root.comops.site
build id 4569617
I20260806 06:26:25.521855 22389 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 06:26:25.522066 22389 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 06:26:25.526882 22399 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:26:25.526891 22394 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:26:25.526933 22396 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:26:25.527047 22389 server_base.cc:1034] running on GCE node
I20260806 06:26:25.527323 22389 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 06:26:25.527546 22389 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 06:26:25.528709 22389 hybrid_clock.cc:648] HybridClock initialized: now 1785997585528699 us; error 27 us; skew 500 ppm
Aug 06 06:26:25 dist-test-slave-4tdj krb5kdc[22249](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997585, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 06:26:25.531245 22389 init.cc:377] Logged in from keytab as kudu/127.21.12.129@KRBTEST.COM (short username kudu)
I20260806 06:26:25.531666 22389 webserver.cc:460] Webserver started at http://127.21.12.129:42677/ using document root <none> and password file <none>
I20260806 06:26:25.531888 22389 fs_manager.cc:359] Metadata directory not provided
I20260806 06:26:25.531931 22389 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 06:26:25.532043 22389 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 06:26:25.533064 22389 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785997581049705-21554-0/minicluster-data/ts-0/data/instance:
uuid: "a41656ddc7174a9e9d33ee1fd315ae03"
format_stamp: "Formatted at 2026-08-06 06:26:25 on dist-test-slave-4tdj"
server_key: "9413953c38eed03777f8f8962419ea33"
server_key_iv: "1c5598b95dcccc47508988a38a72bb5f"
server_key_version: "encryptionkey@0"
I20260806 06:26:25.533388 22389 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785997581049705-21554-0/minicluster-data/ts-0/wal/instance:
uuid: "a41656ddc7174a9e9d33ee1fd315ae03"
format_stamp: "Formatted at 2026-08-06 06:26:25 on dist-test-slave-4tdj"
server_key: "9413953c38eed03777f8f8962419ea33"
server_key_iv: "1c5598b95dcccc47508988a38a72bb5f"
server_key_version: "encryptionkey@0"
I20260806 06:26:25.534637 22389 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260806 06:26:25.535300 22407 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 06:26:25.535426 22389 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260806 06:26:25.535506 22389 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785997581049705-21554-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785997581049705-21554-0/minicluster-data/ts-0/wal
uuid: "a41656ddc7174a9e9d33ee1fd315ae03"
format_stamp: "Formatted at 2026-08-06 06:26:25 on dist-test-slave-4tdj"
server_key: "9413953c38eed03777f8f8962419ea33"
server_key_iv: "1c5598b95dcccc47508988a38a72bb5f"
server_key_version: "encryptionkey@0"
I20260806 06:26:25.535569 22389 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785997581049705-21554-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785997581049705-21554-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785997581049705-21554-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 06:26:25.547742 22389 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 06:26:25.549407 22389 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 06:26:25.549531 22389 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 06:26:25.549731 22389 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 06:26:25.550014 22389 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 06:26:25.550047 22389 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 06:26:25.550068 22389 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 06:26:25.550078 22389 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 06:26:25.555675 22389 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.129:43361
I20260806 06:26:25.555742 22558 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.129:43361 every 8 connection(s)
I20260806 06:26:25.556020 22389 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785997581049705-21554-0/minicluster-data/ts-0/data/info.pb
Aug 06 06:26:25 dist-test-slave-4tdj krb5kdc[22249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997585, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.129@KRBTEST.COM for kudu/127.21.12.190@KRBTEST.COM
I20260806 06:26:25.563839 21554 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHhLhT/build/release/bin/kudu as pid 22389
I20260806 06:26:25.563937 21554 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785997581049705-21554-0/minicluster-data/ts-0/wal/instance
I20260806 06:26:25.564103 21554 external_mini_cluster.cc:1428] Setting key be39bf1612c4fa1d5dd2d2bc0e33c019
I20260806 06:26:25.565264 22386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:25.556714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.129:42253 (local address 127.21.12.190:41629)
0806 06:26:25.556772 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0806 06:26:25.556774 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:26:25.556940 (+ 166us) server_negotiation.cc:407] Connection header received
0806 06:26:25.557976 (+ 1036us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:26:25.557978 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:26:25.557999 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:26:25.558050 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:26:25.558667 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:25.559174 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:25.560133 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:25.560334 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:25.560388 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:25.562409 (+ 2021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:26:25.562411 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:26:25.562411 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 06:26:25.562417 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:26:25.563639 (+ 1222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:26:25.564116 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:26:25.564118 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:26:25.564118 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:26:25.564186 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:26:25.564617 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:26:25.564617 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:26:25.564618 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:26:25.564728 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:26:25.564789 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0806 06:26:25.565060 (+ 271us) server_negotiation.cc:299] Negotiation successful
0806 06:26:25.565158 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260806 06:26:25.565397 22562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:25.556681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:41629 (local address 127.21.12.129:42253)
0806 06:26:25.556825 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0806 06:26:25.556863 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0806 06:26:25.557862 (+ 999us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:26:25.558124 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:26:25.558128 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:26:25.558380 (+ 252us) client_negotiation.cc:189] Negotiated authn=SASL
0806 06:26:25.558571 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:25.558575 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:25.559252 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:25.559254 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:25.560047 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:25.560049 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:25.560401 (+ 352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:25.560402 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:25.560525 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:25.560960 (+ 435us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 06:26:25.560969 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0806 06:26:25.562320 (+ 1351us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 06:26:25.563730 (+ 1410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:26:25.563737 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:26:25.563739 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:26:25.564032 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:26:25.564254 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:26:25.564256 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:26:25.564256 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:26:25.564365 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:26:25.564792 (+ 427us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 06:26:25.564800 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 06:26:25.564933 (+ 133us) client_negotiation.cc:769] Sending connection context
0806 06:26:25.565025 (+ 92us) client_negotiation.cc:240] Negotiation successful
0806 06:26:25.565174 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":69,"threads_started":1}
I20260806 06:26:25.565968 22559 heartbeater.cc:347] Connected to a master server at 127.21.12.190:41629
I20260806 06:26:25.566052 22559 heartbeater.cc:464] Registering TS with master...
I20260806 06:26:25.566218 22559 heartbeater.cc:511] Master 127.21.12.190:41629 requested a full tablet report, sending...
I20260806 06:26:25.566740 22303 ts_manager.cc:194] Registered new tserver with Master: a41656ddc7174a9e9d33ee1fd315ae03 (127.21.12.129:43361)
I20260806 06:26:25.567363 22303 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.12.129@KRBTEST.COM'} at 127.21.12.129:42253
WARNING: no policy specified for kudu/127.21.12.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.12.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.12.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.130.keytab.
Entry for principal kudu/127.21.12.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.130.keytab.
I20260806 06:26:25.588775 21554 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHhLhT/build/release/bin/kudu
/tmp/dist-test-taskFHhLhT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785997581049705-21554-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785997581049705-21554-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785997581049705-21554-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785997581049705-21554-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.12.130:0
--local_ip_for_outbound_sockets=127.21.12.130
--webserver_interface=127.21.12.130
--webserver_port=0
--tserver_master_addrs=127.21.12.190:41629
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.130.keytab
--principal=kudu/127.21.12.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.148:45247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785997581049705-21554-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785997581049705-21554-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785997581049705-21554-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 06:26:25.670223 22566 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 06:26:25.670461 22566 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 06:26:25.670485 22566 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 06:26:25.671638 22566 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 06:26:25.671679 22566 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 06:26:25.671703 22566 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.12.130
I20260806 06:26:25.673238 22566 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.148:45247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785997581049705-21554-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785997581049705-21554-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.130.keytab
--principal=kudu/127.21.12.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.12.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785997581049705-21554-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.12.130
--webserver_port=0
--tserver_master_addrs=127.21.12.190:41629
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22566
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.12.130
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785997581049705-21554-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 05:11:02 UTC on ccycloud.quasar-hvzbbz.root.comops.site
build id 4569617
I20260806 06:26:25.673450 22566 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 06:26:25.673650 22566 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 06:26:25.678560 22571 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:26:25.678565 22576 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:26:25.678601 22566 server_base.cc:1034] running on GCE node
W20260806 06:26:25.678596 22573 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:26:25.678952 22566 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 06:26:25.679191 22566 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 06:26:25.680337 22566 hybrid_clock.cc:648] HybridClock initialized: now 1785997585680324 us; error 30 us; skew 500 ppm
Aug 06 06:26:25 dist-test-slave-4tdj krb5kdc[22249](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997585, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 06:26:25.682866 22566 init.cc:377] Logged in from keytab as kudu/127.21.12.130@KRBTEST.COM (short username kudu)
I20260806 06:26:25.683300 22566 webserver.cc:460] Webserver started at http://127.21.12.130:42915/ using document root <none> and password file <none>
I20260806 06:26:25.683516 22566 fs_manager.cc:359] Metadata directory not provided
I20260806 06:26:25.683570 22566 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 06:26:25.683696 22566 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 06:26:25.684700 22566 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785997581049705-21554-0/minicluster-data/ts-1/data/instance:
uuid: "86536713fa9c4926af1eeaa0bd5f9273"
format_stamp: "Formatted at 2026-08-06 06:26:25 on dist-test-slave-4tdj"
server_key: "41b23e1c1b037308cc8f5ac6f31d95f2"
server_key_iv: "dd2202fdb7fc5115c757a9ad7a39d64a"
server_key_version: "encryptionkey@0"
I20260806 06:26:25.685024 22566 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785997581049705-21554-0/minicluster-data/ts-1/wal/instance:
uuid: "86536713fa9c4926af1eeaa0bd5f9273"
format_stamp: "Formatted at 2026-08-06 06:26:25 on dist-test-slave-4tdj"
server_key: "41b23e1c1b037308cc8f5ac6f31d95f2"
server_key_iv: "dd2202fdb7fc5115c757a9ad7a39d64a"
server_key_version: "encryptionkey@0"
I20260806 06:26:25.686242 22566 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260806 06:26:25.687050 22587 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 06:26:25.687211 22566 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260806 06:26:25.687286 22566 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785997581049705-21554-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785997581049705-21554-0/minicluster-data/ts-1/wal
uuid: "86536713fa9c4926af1eeaa0bd5f9273"
format_stamp: "Formatted at 2026-08-06 06:26:25 on dist-test-slave-4tdj"
server_key: "41b23e1c1b037308cc8f5ac6f31d95f2"
server_key_iv: "dd2202fdb7fc5115c757a9ad7a39d64a"
server_key_version: "encryptionkey@0"
I20260806 06:26:25.687345 22566 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785997581049705-21554-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785997581049705-21554-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785997581049705-21554-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 06:26:25.696518 22566 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 06:26:25.698230 22566 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 06:26:25.698385 22566 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 06:26:25.698599 22566 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 06:26:25.698884 22566 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 06:26:25.698918 22566 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 06:26:25.698945 22566 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 06:26:25.698969 22566 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 06:26:25.704643 22566 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.130:41449
I20260806 06:26:25.704685 22733 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.130:41449 every 8 connection(s)
I20260806 06:26:25.704984 22566 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785997581049705-21554-0/minicluster-data/ts-1/data/info.pb
Aug 06 06:26:25 dist-test-slave-4tdj krb5kdc[22249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997585, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.130@KRBTEST.COM for kudu/127.21.12.190@KRBTEST.COM
I20260806 06:26:25.713467 21554 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHhLhT/build/release/bin/kudu as pid 22566
I20260806 06:26:25.713522 22386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:25.705680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.130:49159 (local address 127.21.12.190:41629)
0806 06:26:25.705737 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0806 06:26:25.705739 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:26:25.705896 (+ 157us) server_negotiation.cc:407] Connection header received
0806 06:26:25.706841 (+ 945us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:26:25.706842 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:26:25.706856 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:26:25.706910 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:26:25.707521 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:25.708012 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:25.708966 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:25.709158 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:25.709214 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:25.711187 (+ 1973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:26:25.711189 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:26:25.711189 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 06:26:25.711198 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:26:25.712296 (+ 1098us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:26:25.712658 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:26:25.712659 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:26:25.712659 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:26:25.712722 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:26:25.712985 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:26:25.712986 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:26:25.712987 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:26:25.713092 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:26:25.713144 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0806 06:26:25.713354 (+ 210us) server_negotiation.cc:299] Negotiation successful
0806 06:26:25.713432 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260806 06:26:25.713555 21554 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785997581049705-21554-0/minicluster-data/ts-1/wal/instance
I20260806 06:26:25.713645 22736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:25.705639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:41629 (local address 127.21.12.130:49159)
0806 06:26:25.705777 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0806 06:26:25.705821 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0806 06:26:25.706739 (+ 918us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:26:25.706984 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:26:25.706989 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:26:25.707253 (+ 264us) client_negotiation.cc:189] Negotiated authn=SASL
0806 06:26:25.707432 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:25.707436 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:25.708092 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:25.708093 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:25.708883 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:25.708885 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:25.709226 (+ 341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:25.709227 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:25.709339 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:25.709780 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 06:26:25.709788 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0806 06:26:25.711094 (+ 1306us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 06:26:25.712367 (+ 1273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:26:25.712373 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:26:25.712375 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:26:25.712578 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:26:25.712807 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:26:25.712808 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:26:25.712808 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:26:25.712925 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:26:25.713142 (+ 217us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 06:26:25.713145 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 06:26:25.713249 (+ 104us) client_negotiation.cc:769] Sending connection context
0806 06:26:25.713326 (+ 77us) client_negotiation.cc:240] Negotiation successful
0806 06:26:25.713445 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":64,"threads_started":1}
I20260806 06:26:25.713709 21554 external_mini_cluster.cc:1428] Setting key 6b98143631295922e6a570ecd937bfd8
I20260806 06:26:25.714159 22734 heartbeater.cc:347] Connected to a master server at 127.21.12.190:41629
I20260806 06:26:25.714244 22734 heartbeater.cc:464] Registering TS with master...
I20260806 06:26:25.714432 22734 heartbeater.cc:511] Master 127.21.12.190:41629 requested a full tablet report, sending...
I20260806 06:26:25.714818 22302 ts_manager.cc:194] Registered new tserver with Master: 86536713fa9c4926af1eeaa0bd5f9273 (127.21.12.130:41449)
I20260806 06:26:25.715185 22302 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.12.130@KRBTEST.COM'} at 127.21.12.130:49159
WARNING: no policy specified for kudu/127.21.12.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.12.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.12.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.131.keytab.
Entry for principal kudu/127.21.12.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.131.keytab.
I20260806 06:26:25.738979 21554 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHhLhT/build/release/bin/kudu
/tmp/dist-test-taskFHhLhT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785997581049705-21554-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785997581049705-21554-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785997581049705-21554-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785997581049705-21554-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.12.131:0
--local_ip_for_outbound_sockets=127.21.12.131
--webserver_interface=127.21.12.131
--webserver_port=0
--tserver_master_addrs=127.21.12.190:41629
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.131.keytab
--principal=kudu/127.21.12.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.148:45247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785997581049705-21554-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785997581049705-21554-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785997581049705-21554-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 06:26:25.818766 22740 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 06:26:25.818969 22740 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 06:26:25.819000 22740 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 06:26:25.820135 22740 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 06:26:25.820174 22740 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 06:26:25.820209 22740 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.12.131
I20260806 06:26:25.821708 22740 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.148:45247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785997581049705-21554-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785997581049705-21554-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.131.keytab
--principal=kudu/127.21.12.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.12.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785997581049705-21554-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.12.131
--webserver_port=0
--tserver_master_addrs=127.21.12.190:41629
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22740
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.12.131
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785997581049705-21554-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 05:11:02 UTC on ccycloud.quasar-hvzbbz.root.comops.site
build id 4569617
I20260806 06:26:25.821934 22740 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 06:26:25.822136 22740 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 06:26:25.826948 22749 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:26:25.826946 22748 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:26:25.827090 22740 server_base.cc:1034] running on GCE node
W20260806 06:26:25.826964 22752 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:26:25.827355 22740 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 06:26:25.827564 22740 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 06:26:25.828733 22740 hybrid_clock.cc:648] HybridClock initialized: now 1785997585828715 us; error 34 us; skew 500 ppm
Aug 06 06:26:25 dist-test-slave-4tdj krb5kdc[22249](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997585, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 06:26:25.831138 22740 init.cc:377] Logged in from keytab as kudu/127.21.12.131@KRBTEST.COM (short username kudu)
I20260806 06:26:25.831522 22740 webserver.cc:460] Webserver started at http://127.21.12.131:33051/ using document root <none> and password file <none>
I20260806 06:26:25.831729 22740 fs_manager.cc:359] Metadata directory not provided
I20260806 06:26:25.831771 22740 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 06:26:25.831878 22740 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 06:26:25.832880 22740 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785997581049705-21554-0/minicluster-data/ts-2/data/instance:
uuid: "2ab6625ca0be47dba63aaac73794d707"
format_stamp: "Formatted at 2026-08-06 06:26:25 on dist-test-slave-4tdj"
server_key: "32423e108c3ed855490b6dba49802efe"
server_key_iv: "80302a247cc40124c1700bd079701ccd"
server_key_version: "encryptionkey@0"
I20260806 06:26:25.833202 22740 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785997581049705-21554-0/minicluster-data/ts-2/wal/instance:
uuid: "2ab6625ca0be47dba63aaac73794d707"
format_stamp: "Formatted at 2026-08-06 06:26:25 on dist-test-slave-4tdj"
server_key: "32423e108c3ed855490b6dba49802efe"
server_key_iv: "80302a247cc40124c1700bd079701ccd"
server_key_version: "encryptionkey@0"
I20260806 06:26:25.834403 22740 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260806 06:26:25.835067 22762 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 06:26:25.835212 22740 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260806 06:26:25.835282 22740 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785997581049705-21554-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785997581049705-21554-0/minicluster-data/ts-2/wal
uuid: "2ab6625ca0be47dba63aaac73794d707"
format_stamp: "Formatted at 2026-08-06 06:26:25 on dist-test-slave-4tdj"
server_key: "32423e108c3ed855490b6dba49802efe"
server_key_iv: "80302a247cc40124c1700bd079701ccd"
server_key_version: "encryptionkey@0"
I20260806 06:26:25.835341 22740 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785997581049705-21554-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785997581049705-21554-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785997581049705-21554-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 06:26:25.848904 22740 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 06:26:25.850838 22740 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 06:26:25.850970 22740 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 06:26:25.851191 22740 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 06:26:25.851476 22740 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 06:26:25.851511 22740 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 06:26:25.851540 22740 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 06:26:25.851564 22740 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 06:26:25.857077 22740 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.131:40109
I20260806 06:26:25.857121 22918 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.131:40109 every 8 connection(s)
I20260806 06:26:25.857432 22740 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785997581049705-21554-0/minicluster-data/ts-2/data/info.pb
Aug 06 06:26:25 dist-test-slave-4tdj krb5kdc[22249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997585, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.131@KRBTEST.COM for kudu/127.21.12.190@KRBTEST.COM
I20260806 06:26:25.863773 21554 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHhLhT/build/release/bin/kudu as pid 22740
I20260806 06:26:25.863866 21554 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785997581049705-21554-0/minicluster-data/ts-2/wal/instance
I20260806 06:26:25.864022 21554 external_mini_cluster.cc:1428] Setting key 1868143aa614f27f6321479063aa04d4
I20260806 06:26:25.866637 22386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:25.858141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.131:48737 (local address 127.21.12.190:41629)
0806 06:26:25.858191 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0806 06:26:25.858193 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:26:25.858448 (+ 255us) server_negotiation.cc:407] Connection header received
0806 06:26:25.859365 (+ 917us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:26:25.859366 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:26:25.859380 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:26:25.859432 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:26:25.860022 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:25.860522 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:25.861492 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:25.861697 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:25.861751 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:25.864071 (+ 2320us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:26:25.864073 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:26:25.864074 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 06:26:25.864082 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:26:25.865455 (+ 1373us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:26:25.865805 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:26:25.865806 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:26:25.865806 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:26:25.865853 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:26:25.866033 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:26:25.866033 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:26:25.866034 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:26:25.866140 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:26:25.866184 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0806 06:26:25.866510 (+ 326us) server_negotiation.cc:299] Negotiation successful
0806 06:26:25.866574 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260806 06:26:25.866794 22922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:25.858110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:41629 (local address 127.21.12.131:48737)
0806 06:26:25.858328 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0806 06:26:25.858373 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0806 06:26:25.859257 (+ 884us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:26:25.859501 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:26:25.859505 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:26:25.859755 (+ 250us) client_negotiation.cc:189] Negotiated authn=SASL
0806 06:26:25.859933 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:25.859937 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:25.860602 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:25.860603 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:25.861410 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:25.861413 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:25.861764 (+ 351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:25.861765 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:25.861888 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:25.862448 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 06:26:25.862457 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0806 06:26:25.863973 (+ 1516us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 06:26:25.865532 (+ 1559us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:26:25.865537 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:26:25.865539 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:26:25.865750 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:26:25.865897 (+ 147us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:26:25.865898 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:26:25.865898 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:26:25.865984 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:26:25.866181 (+ 197us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 06:26:25.866183 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 06:26:25.866388 (+ 205us) client_negotiation.cc:769] Sending connection context
0806 06:26:25.866487 (+ 99us) client_negotiation.cc:240] Negotiation successful
0806 06:26:25.866610 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":124,"threads_started":1}
I20260806 06:26:25.867233 22919 heartbeater.cc:347] Connected to a master server at 127.21.12.190:41629
I20260806 06:26:25.867321 22919 heartbeater.cc:464] Registering TS with master...
I20260806 06:26:25.867487 22919 heartbeater.cc:511] Master 127.21.12.190:41629 requested a full tablet report, sending...
I20260806 06:26:25.867812 22302 ts_manager.cc:194] Registered new tserver with Master: 2ab6625ca0be47dba63aaac73794d707 (127.21.12.131:40109)
I20260806 06:26:25.868173 22302 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.12.131@KRBTEST.COM'} at 127.21.12.131:48737
I20260806 06:26:25.875731 21554 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 06:26:25.882962 22386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:25.877355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39662 (local address 127.21.12.190:41629)
0806 06:26:25.877418 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0806 06:26:25.877420 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:26:25.877491 (+ 71us) server_negotiation.cc:407] Connection header received
0806 06:26:25.877573 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:26:25.877573 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:26:25.877582 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:26:25.877618 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:26:25.878154 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:25.878710 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:25.879413 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:25.879615 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:25.879667 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:25.880745 (+ 1078us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:26:25.880747 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:26:25.880747 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 06:26:25.880753 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:26:25.881915 (+ 1162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:26:25.882254 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:26:25.882255 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:26:25.882255 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:26:25.882351 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:26:25.882515 (+ 164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:26:25.882516 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:26:25.882517 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:26:25.882668 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:26:25.882747 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0806 06:26:25.882827 (+ 80us) server_negotiation.cc:299] Negotiation successful
0806 06:26:25.882888 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260806 06:26:25.885234 22302 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39662:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 06:26:25.886811 22302 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 06:26:25.893270 22930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:25.888872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.131:40109 (local address 127.0.0.1:56444)
0806 06:26:25.889708 (+ 836us) negotiation.cc:107] Waiting for socket to connect
0806 06:26:25.889718 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0806 06:26:25.889846 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:26:25.890206 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:26:25.890207 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:26:25.890209 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 06:26:25.890248 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:25.890249 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:25.891273 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:25.891275 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:25.892337 (+ 1062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:25.892338 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:25.893083 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:25.893084 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:25.893136 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:25.893140 (+ 4us) client_negotiation.cc:769] Sending connection context
0806 06:26:25.893181 (+ 41us) client_negotiation.cc:240] Negotiation successful
0806 06:26:25.893210 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":822,"thread_start_us":98,"threads_started":1}
I20260806 06:26:25.893311 22934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:25.888998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60382 (local address 127.21.12.130:41449)
0806 06:26:25.889529 (+ 531us) server_negotiation.cc:206] Beginning negotiation
0806 06:26:25.889533 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 06:26:25.889543 (+ 10us) server_negotiation.cc:407] Connection header received
0806 06:26:25.889597 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:26:25.889599 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:26:25.889615 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:26:25.889705 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 06:26:25.889882 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:25.891064 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:25.892188 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:25.892879 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:25.892956 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:25.893014 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0806 06:26:25.893156 (+ 142us) server_negotiation.cc:299] Negotiation successful
0806 06:26:25.893198 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":474,"thread_start_us":157,"threads_started":1}
I20260806 06:26:25.893509 22935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:25.889136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41074 (local address 127.21.12.129:43361)
0806 06:26:25.889370 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0806 06:26:25.889375 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 06:26:25.889385 (+ 10us) server_negotiation.cc:407] Connection header received
0806 06:26:25.889442 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:26:25.889444 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:26:25.889458 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:26:25.889523 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 06:26:25.889742 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:25.890910 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:25.892396 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:25.893091 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:25.893149 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:25.893199 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0806 06:26:25.893399 (+ 200us) server_negotiation.cc:299] Negotiation successful
0806 06:26:25.893441 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":183,"thread_start_us":85,"threads_started":1}
I20260806 06:26:25.893270 22932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:25.888865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.130:41449 (local address 127.0.0.1:60382)
0806 06:26:25.889030 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0806 06:26:25.889084 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0806 06:26:25.889150 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:26:25.889738 (+ 588us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:26:25.889739 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:26:25.889740 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 06:26:25.889792 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:25.889794 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:25.891220 (+ 1426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:25.891221 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:25.892118 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:25.892121 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:25.892971 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:25.892972 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:25.893049 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:25.893063 (+ 14us) client_negotiation.cc:769] Sending connection context
0806 06:26:25.893133 (+ 70us) client_negotiation.cc:240] Negotiation successful
0806 06:26:25.893185 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":56,"threads_started":1}
I20260806 06:26:25.893668 22936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:25.889202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56444 (local address 127.21.12.131:40109)
0806 06:26:25.889370 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0806 06:26:25.889375 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 06:26:25.889829 (+ 454us) server_negotiation.cc:407] Connection header received
0806 06:26:25.889970 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:26:25.889973 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:26:25.889989 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:26:25.890058 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 06:26:25.890340 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:25.891130 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:25.892416 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:25.892918 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:25.892984 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:25.893051 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0806 06:26:25.893508 (+ 457us) server_negotiation.cc:299] Negotiation successful
0806 06:26:25.893552 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":83,"threads_started":1}
I20260806 06:26:25.893721 22933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:25.888865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.129:43361 (local address 127.0.0.1:41074)
0806 06:26:25.889294 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0806 06:26:25.889305 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0806 06:26:25.889357 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:26:25.889570 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:26:25.889573 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:26:25.889576 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 06:26:25.889646 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:25.889648 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:25.891023 (+ 1375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:25.891024 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:25.892305 (+ 1281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:25.892308 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:25.893235 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:25.893236 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:25.893297 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:25.893301 (+ 4us) client_negotiation.cc:769] Sending connection context
0806 06:26:25.893613 (+ 312us) client_negotiation.cc:240] Negotiation successful
0806 06:26:25.893658 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":365,"mutex_wait_us":5,"thread_start_us":64,"threads_started":1}
I20260806 06:26:25.894387 22646 tablet_service.cc:1467] Processing CreateTablet for tablet 5826563206774a97a436464d671b8955 (DEFAULT_TABLE table=test-table [id=2a789d1795154c518211222bceb2778c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 06:26:25.894387 22822 tablet_service.cc:1467] Processing CreateTablet for tablet 5826563206774a97a436464d671b8955 (DEFAULT_TABLE table=test-table [id=2a789d1795154c518211222bceb2778c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 06:26:25.894763 22822 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5826563206774a97a436464d671b8955. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 06:26:25.894763 22646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5826563206774a97a436464d671b8955. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 06:26:25.898729 22468 tablet_service.cc:1467] Processing CreateTablet for tablet 5826563206774a97a436464d671b8955 (DEFAULT_TABLE table=test-table [id=2a789d1795154c518211222bceb2778c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 06:26:25.899057 22937 tablet_bootstrap.cc:492] T 5826563206774a97a436464d671b8955 P 2ab6625ca0be47dba63aaac73794d707: Bootstrap starting.
I20260806 06:26:25.899120 22468 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5826563206774a97a436464d671b8955. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 06:26:25.899307 22938 tablet_bootstrap.cc:492] T 5826563206774a97a436464d671b8955 P 86536713fa9c4926af1eeaa0bd5f9273: Bootstrap starting.
I20260806 06:26:25.900274 22937 tablet_bootstrap.cc:654] T 5826563206774a97a436464d671b8955 P 2ab6625ca0be47dba63aaac73794d707: Neither blocks nor log segments found. Creating new log.
I20260806 06:26:25.900456 22938 tablet_bootstrap.cc:654] T 5826563206774a97a436464d671b8955 P 86536713fa9c4926af1eeaa0bd5f9273: Neither blocks nor log segments found. Creating new log.
I20260806 06:26:25.901005 22937 log.cc:824] T 5826563206774a97a436464d671b8955 P 2ab6625ca0be47dba63aaac73794d707: Log is configured to *not* fsync() on all Append() calls
I20260806 06:26:25.901314 22938 log.cc:824] T 5826563206774a97a436464d671b8955 P 86536713fa9c4926af1eeaa0bd5f9273: Log is configured to *not* fsync() on all Append() calls
I20260806 06:26:25.902043 22938 tablet_bootstrap.cc:492] T 5826563206774a97a436464d671b8955 P 86536713fa9c4926af1eeaa0bd5f9273: No bootstrap required, opened a new log
I20260806 06:26:25.902107 22938 ts_tablet_manager.cc:1397] T 5826563206774a97a436464d671b8955 P 86536713fa9c4926af1eeaa0bd5f9273: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260806 06:26:25.902156 22937 tablet_bootstrap.cc:492] T 5826563206774a97a436464d671b8955 P 2ab6625ca0be47dba63aaac73794d707: No bootstrap required, opened a new log
I20260806 06:26:25.902242 22937 ts_tablet_manager.cc:1397] T 5826563206774a97a436464d671b8955 P 2ab6625ca0be47dba63aaac73794d707: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260806 06:26:25.903504 22942 tablet_bootstrap.cc:492] T 5826563206774a97a436464d671b8955 P a41656ddc7174a9e9d33ee1fd315ae03: Bootstrap starting.
I20260806 06:26:25.904691 22942 tablet_bootstrap.cc:654] T 5826563206774a97a436464d671b8955 P a41656ddc7174a9e9d33ee1fd315ae03: Neither blocks nor log segments found. Creating new log.
I20260806 06:26:25.904726 22937 raft_consensus.cc:348] T 5826563206774a97a436464d671b8955 P 2ab6625ca0be47dba63aaac73794d707 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86536713fa9c4926af1eeaa0bd5f9273" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 41449 } } peers { permanent_uuid: "2ab6625ca0be47dba63aaac73794d707" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 40109 } } peers { permanent_uuid: "a41656ddc7174a9e9d33ee1fd315ae03" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 43361 } }
I20260806 06:26:25.904747 22938 raft_consensus.cc:348] T 5826563206774a97a436464d671b8955 P 86536713fa9c4926af1eeaa0bd5f9273 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86536713fa9c4926af1eeaa0bd5f9273" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 41449 } } peers { permanent_uuid: "2ab6625ca0be47dba63aaac73794d707" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 40109 } } peers { permanent_uuid: "a41656ddc7174a9e9d33ee1fd315ae03" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 43361 } }
I20260806 06:26:25.904928 22937 raft_consensus.cc:374] T 5826563206774a97a436464d671b8955 P 2ab6625ca0be47dba63aaac73794d707 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 06:26:25.904955 22938 raft_consensus.cc:374] T 5826563206774a97a436464d671b8955 P 86536713fa9c4926af1eeaa0bd5f9273 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 06:26:25.904978 22937 raft_consensus.cc:729] T 5826563206774a97a436464d671b8955 P 2ab6625ca0be47dba63aaac73794d707 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ab6625ca0be47dba63aaac73794d707, State: Initialized, Role: FOLLOWER
I20260806 06:26:25.904997 22938 raft_consensus.cc:729] T 5826563206774a97a436464d671b8955 P 86536713fa9c4926af1eeaa0bd5f9273 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86536713fa9c4926af1eeaa0bd5f9273, State: Initialized, Role: FOLLOWER
I20260806 06:26:25.905117 22938 consensus_queue.cc:260] T 5826563206774a97a436464d671b8955 P 86536713fa9c4926af1eeaa0bd5f9273 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86536713fa9c4926af1eeaa0bd5f9273" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 41449 } } peers { permanent_uuid: "2ab6625ca0be47dba63aaac73794d707" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 40109 } } peers { permanent_uuid: "a41656ddc7174a9e9d33ee1fd315ae03" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 43361 } }
I20260806 06:26:25.905140 22937 consensus_queue.cc:260] T 5826563206774a97a436464d671b8955 P 2ab6625ca0be47dba63aaac73794d707 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86536713fa9c4926af1eeaa0bd5f9273" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 41449 } } peers { permanent_uuid: "2ab6625ca0be47dba63aaac73794d707" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 40109 } } peers { permanent_uuid: "a41656ddc7174a9e9d33ee1fd315ae03" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 43361 } }
I20260806 06:26:25.905433 22937 ts_tablet_manager.cc:1428] T 5826563206774a97a436464d671b8955 P 2ab6625ca0be47dba63aaac73794d707: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260806 06:26:25.905458 22938 ts_tablet_manager.cc:1428] T 5826563206774a97a436464d671b8955 P 86536713fa9c4926af1eeaa0bd5f9273: Time spent starting tablet: real 0.003s user 0.000s sys 0.004s
I20260806 06:26:25.905465 22734 heartbeater.cc:503] Master 127.21.12.190:41629 was elected leader, sending a full tablet report...
I20260806 06:26:25.905545 22919 heartbeater.cc:503] Master 127.21.12.190:41629 was elected leader, sending a full tablet report...
I20260806 06:26:25.905658 22942 log.cc:824] T 5826563206774a97a436464d671b8955 P a41656ddc7174a9e9d33ee1fd315ae03: Log is configured to *not* fsync() on all Append() calls
I20260806 06:26:25.906541 22942 tablet_bootstrap.cc:492] T 5826563206774a97a436464d671b8955 P a41656ddc7174a9e9d33ee1fd315ae03: No bootstrap required, opened a new log
I20260806 06:26:25.906610 22942 ts_tablet_manager.cc:1397] T 5826563206774a97a436464d671b8955 P a41656ddc7174a9e9d33ee1fd315ae03: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20260806 06:26:25.909027 22942 raft_consensus.cc:348] T 5826563206774a97a436464d671b8955 P a41656ddc7174a9e9d33ee1fd315ae03 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86536713fa9c4926af1eeaa0bd5f9273" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 41449 } } peers { permanent_uuid: "2ab6625ca0be47dba63aaac73794d707" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 40109 } } peers { permanent_uuid: "a41656ddc7174a9e9d33ee1fd315ae03" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 43361 } }
I20260806 06:26:25.909224 22942 raft_consensus.cc:374] T 5826563206774a97a436464d671b8955 P a41656ddc7174a9e9d33ee1fd315ae03 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 06:26:25.909250 22942 raft_consensus.cc:729] T 5826563206774a97a436464d671b8955 P a41656ddc7174a9e9d33ee1fd315ae03 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a41656ddc7174a9e9d33ee1fd315ae03, State: Initialized, Role: FOLLOWER
I20260806 06:26:25.909329 22942 consensus_queue.cc:260] T 5826563206774a97a436464d671b8955 P a41656ddc7174a9e9d33ee1fd315ae03 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86536713fa9c4926af1eeaa0bd5f9273" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 41449 } } peers { permanent_uuid: "2ab6625ca0be47dba63aaac73794d707" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 40109 } } peers { permanent_uuid: "a41656ddc7174a9e9d33ee1fd315ae03" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 43361 } }
I20260806 06:26:25.909538 22942 ts_tablet_manager.cc:1428] T 5826563206774a97a436464d671b8955 P a41656ddc7174a9e9d33ee1fd315ae03: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260806 06:26:25.909607 22559 heartbeater.cc:503] Master 127.21.12.190:41629 was elected leader, sending a full tablet report...
W20260806 06:26:25.955600 22735 tablet.cc:2369] T 5826563206774a97a436464d671b8955 P 86536713fa9c4926af1eeaa0bd5f9273: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 06:26:25.966758 22947 raft_consensus.cc:482] T 5826563206774a97a436464d671b8955 P a41656ddc7174a9e9d33ee1fd315ae03 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 06:26:25.966915 22947 raft_consensus.cc:504] T 5826563206774a97a436464d671b8955 P a41656ddc7174a9e9d33ee1fd315ae03 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86536713fa9c4926af1eeaa0bd5f9273" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 41449 } } peers { permanent_uuid: "2ab6625ca0be47dba63aaac73794d707" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 40109 } } peers { permanent_uuid: "a41656ddc7174a9e9d33ee1fd315ae03" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 43361 } }
I20260806 06:26:25.967372 22947 leader_election.cc:288] T 5826563206774a97a436464d671b8955 P a41656ddc7174a9e9d33ee1fd315ae03 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 86536713fa9c4926af1eeaa0bd5f9273 (127.21.12.130:41449), 2ab6625ca0be47dba63aaac73794d707 (127.21.12.131:40109)
I20260806 06:26:25.970319 22934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:25.967633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.129:38261 (local address 127.21.12.130:41449)
0806 06:26:25.967691 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0806 06:26:25.967693 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:26:25.967701 (+ 8us) server_negotiation.cc:407] Connection header received
0806 06:26:25.967773 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:26:25.967774 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:26:25.967785 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:26:25.967833 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 06:26:25.968089 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:25.968555 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:25.969501 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:25.969920 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:25.969986 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:25.969996 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0806 06:26:25.970195 (+ 199us) server_negotiation.cc:299] Negotiation successful
0806 06:26:25.970236 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260806 06:26:25.970317 22562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:25.967579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.130:41449 (local address 127.21.12.129:38261)
0806 06:26:25.967650 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0806 06:26:25.967663 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 06:26:25.967720 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:26:25.967846 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:26:25.967847 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:26:25.967848 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 06:26:25.968018 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:25.968021 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:25.968638 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:25.968639 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:25.969420 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:25.969422 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:25.970001 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:25.970002 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:25.970127 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:25.970131 (+ 4us) client_negotiation.cc:769] Sending connection context
0806 06:26:25.970180 (+ 49us) client_negotiation.cc:240] Negotiation successful
0806 06:26:25.970220 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260806 06:26:25.970665 22669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5826563206774a97a436464d671b8955" candidate_uuid: "a41656ddc7174a9e9d33ee1fd315ae03" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86536713fa9c4926af1eeaa0bd5f9273" is_pre_election: true
I20260806 06:26:25.970822 22936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:25.967687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.129:53419 (local address 127.21.12.131:40109)
0806 06:26:25.967742 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0806 06:26:25.967745 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 06:26:25.967989 (+ 244us) server_negotiation.cc:407] Connection header received
0806 06:26:25.968044 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:26:25.968045 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:26:25.968055 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:26:25.968101 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 06:26:25.968312 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:25.968765 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:25.969898 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:25.970352 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:25.970421 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:25.970430 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0806 06:26:25.970730 (+ 300us) server_negotiation.cc:299] Negotiation successful
0806 06:26:25.970764 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260806 06:26:25.970837 22949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:25.967580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.131:40109 (local address 127.21.12.129:53419)
0806 06:26:25.967909 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0806 06:26:25.967919 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0806 06:26:25.967988 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:26:25.968113 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:26:25.968114 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:26:25.968116 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 06:26:25.968243 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:25.968245 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:25.968845 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:25.968847 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:25.969816 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:25.969818 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:25.970534 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:25.970536 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:25.970636 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:25.970641 (+ 5us) client_negotiation.cc:769] Sending connection context
0806 06:26:25.970702 (+ 61us) client_negotiation.cc:240] Negotiation successful
0806 06:26:25.970741 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"mutex_wait_us":81,"thread_start_us":105,"threads_started":1}
I20260806 06:26:25.970849 22669 raft_consensus.cc:2454] T 5826563206774a97a436464d671b8955 P 86536713fa9c4926af1eeaa0bd5f9273 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a41656ddc7174a9e9d33ee1fd315ae03 in term 0.
I20260806 06:26:25.971091 22409 leader_election.cc:302] T 5826563206774a97a436464d671b8955 P a41656ddc7174a9e9d33ee1fd315ae03 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 86536713fa9c4926af1eeaa0bd5f9273, a41656ddc7174a9e9d33ee1fd315ae03; no voters:
I20260806 06:26:25.971269 22947 raft_consensus.cc:2781] T 5826563206774a97a436464d671b8955 P a41656ddc7174a9e9d33ee1fd315ae03 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 06:26:25.971325 22947 raft_consensus.cc:482] T 5826563206774a97a436464d671b8955 P a41656ddc7174a9e9d33ee1fd315ae03 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 06:26:25.971261 22851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5826563206774a97a436464d671b8955" candidate_uuid: "a41656ddc7174a9e9d33ee1fd315ae03" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ab6625ca0be47dba63aaac73794d707" is_pre_election: true
I20260806 06:26:25.971356 22947 raft_consensus.cc:3037] T 5826563206774a97a436464d671b8955 P a41656ddc7174a9e9d33ee1fd315ae03 [term 0 FOLLOWER]: Advancing to term 1
I20260806 06:26:25.971410 22851 raft_consensus.cc:2454] T 5826563206774a97a436464d671b8955 P 2ab6625ca0be47dba63aaac73794d707 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a41656ddc7174a9e9d33ee1fd315ae03 in term 0.
I20260806 06:26:25.972299 22947 raft_consensus.cc:504] T 5826563206774a97a436464d671b8955 P a41656ddc7174a9e9d33ee1fd315ae03 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86536713fa9c4926af1eeaa0bd5f9273" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 41449 } } peers { permanent_uuid: "2ab6625ca0be47dba63aaac73794d707" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 40109 } } peers { permanent_uuid: "a41656ddc7174a9e9d33ee1fd315ae03" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 43361 } }
I20260806 06:26:25.972425 22947 leader_election.cc:288] T 5826563206774a97a436464d671b8955 P a41656ddc7174a9e9d33ee1fd315ae03 [CANDIDATE]: Term 1 election: Requested vote from peers 86536713fa9c4926af1eeaa0bd5f9273 (127.21.12.130:41449), 2ab6625ca0be47dba63aaac73794d707 (127.21.12.131:40109)
I20260806 06:26:25.972590 22669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5826563206774a97a436464d671b8955" candidate_uuid: "a41656ddc7174a9e9d33ee1fd315ae03" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86536713fa9c4926af1eeaa0bd5f9273"
I20260806 06:26:25.972631 22851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5826563206774a97a436464d671b8955" candidate_uuid: "a41656ddc7174a9e9d33ee1fd315ae03" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ab6625ca0be47dba63aaac73794d707"
I20260806 06:26:25.972671 22669 raft_consensus.cc:3037] T 5826563206774a97a436464d671b8955 P 86536713fa9c4926af1eeaa0bd5f9273 [term 0 FOLLOWER]: Advancing to term 1
I20260806 06:26:25.972695 22851 raft_consensus.cc:3037] T 5826563206774a97a436464d671b8955 P 2ab6625ca0be47dba63aaac73794d707 [term 0 FOLLOWER]: Advancing to term 1
I20260806 06:26:25.973845 22669 raft_consensus.cc:2454] T 5826563206774a97a436464d671b8955 P 86536713fa9c4926af1eeaa0bd5f9273 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a41656ddc7174a9e9d33ee1fd315ae03 in term 1.
I20260806 06:26:25.974007 22409 leader_election.cc:302] T 5826563206774a97a436464d671b8955 P a41656ddc7174a9e9d33ee1fd315ae03 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 86536713fa9c4926af1eeaa0bd5f9273, a41656ddc7174a9e9d33ee1fd315ae03; no voters:
I20260806 06:26:25.974056 22851 raft_consensus.cc:2454] T 5826563206774a97a436464d671b8955 P 2ab6625ca0be47dba63aaac73794d707 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a41656ddc7174a9e9d33ee1fd315ae03 in term 1.
I20260806 06:26:25.974120 22947 raft_consensus.cc:2781] T 5826563206774a97a436464d671b8955 P a41656ddc7174a9e9d33ee1fd315ae03 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 06:26:25.974323 22947 raft_consensus.cc:686] T 5826563206774a97a436464d671b8955 P a41656ddc7174a9e9d33ee1fd315ae03 [term 1 LEADER]: Becoming Leader. State: Replica: a41656ddc7174a9e9d33ee1fd315ae03, State: Running, Role: LEADER
I20260806 06:26:25.974440 22947 consensus_queue.cc:237] T 5826563206774a97a436464d671b8955 P a41656ddc7174a9e9d33ee1fd315ae03 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86536713fa9c4926af1eeaa0bd5f9273" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 41449 } } peers { permanent_uuid: "2ab6625ca0be47dba63aaac73794d707" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 40109 } } peers { permanent_uuid: "a41656ddc7174a9e9d33ee1fd315ae03" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 43361 } }
I20260806 06:26:25.975281 22303 catalog_manager.cc:5495] T 5826563206774a97a436464d671b8955 P a41656ddc7174a9e9d33ee1fd315ae03 reported cstate change: term changed from 0 to 1, leader changed from <none> to a41656ddc7174a9e9d33ee1fd315ae03 (127.21.12.129). New cstate: current_term: 1 leader_uuid: "a41656ddc7174a9e9d33ee1fd315ae03" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86536713fa9c4926af1eeaa0bd5f9273" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 41449 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2ab6625ca0be47dba63aaac73794d707" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 40109 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a41656ddc7174a9e9d33ee1fd315ae03" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 43361 } health_report { overall_health: HEALTHY } } }
Aug 06 06:26:25 dist-test-slave-4tdj krb5kdc[22249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997585, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.12.129@KRBTEST.COM
I20260806 06:26:25.991005 22935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:25.983558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41084 (local address 127.21.12.129:43361)
0806 06:26:25.983625 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0806 06:26:25.983626 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0806 06:26:25.983791 (+ 165us) server_negotiation.cc:407] Connection header received
0806 06:26:25.983814 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:26:25.983814 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:26:25.983824 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:26:25.983865 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:26:25.984472 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:25.984952 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:25.985682 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:25.985875 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:25.985936 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:25.988276 (+ 2340us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:26:25.988286 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:26:25.988286 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 06:26:25.988293 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:26:25.990033 (+ 1740us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:26:25.990383 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:26:25.990386 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:26:25.990387 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:26:25.990445 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:26:25.990583 (+ 138us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:26:25.990584 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:26:25.990584 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:26:25.990710 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:26:25.990824 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0806 06:26:25.990880 (+ 56us) server_negotiation.cc:299] Negotiation successful
0806 06:26:25.990940 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
Aug 06 06:26:26 dist-test-slave-4tdj krb5kdc[22249](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997586, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 06 06:26:26 dist-test-slave-4tdj krb5kdc[22249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997586, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.21.12.129@KRBTEST.COM
I20260806 06:26:26.014374 22935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:26.007963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41100 (local address 127.21.12.129:43361)
0806 06:26:26.008048 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0806 06:26:26.008050 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:26:26.008094 (+ 44us) server_negotiation.cc:407] Connection header received
0806 06:26:26.008148 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:26:26.008149 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:26:26.008158 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:26:26.008206 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:26:26.008682 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:26.009158 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:26.009805 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:26.009974 (+ 169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:26.010015 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:26.012155 (+ 2140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:26:26.012157 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:26:26.012157 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 06:26:26.012163 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:26:26.013388 (+ 1225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:26:26.013714 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:26:26.013715 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:26:26.013715 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:26:26.013765 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:26:26.013902 (+ 137us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:26:26.013902 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:26:26.013902 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:26:26.013984 (+ 82us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:26:26.014058 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0806 06:26:26.014189 (+ 131us) server_negotiation.cc:299] Negotiation successful
0806 06:26:26.014255 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
Aug 06 06:26:26 dist-test-slave-4tdj krb5kdc[22249](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997586, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 06 06:26:26 dist-test-slave-4tdj krb5kdc[22249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997586, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.12.129@KRBTEST.COM
I20260806 06:26:26.036543 22935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:26.030096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41108 (local address 127.21.12.129:43361)
0806 06:26:26.030143 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0806 06:26:26.030145 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:26:26.030194 (+ 49us) server_negotiation.cc:407] Connection header received
0806 06:26:26.030237 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:26:26.030238 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:26:26.030246 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:26:26.030340 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:26:26.030833 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:26.031296 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:26.032004 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:26.032188 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:26.032232 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:26.034423 (+ 2191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:26:26.034425 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:26:26.034425 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 06:26:26.034431 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:26:26.035537 (+ 1106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:26:26.035841 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:26:26.035842 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:26:26.035843 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:26:26.035900 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:26:26.036047 (+ 147us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:26:26.036048 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:26:26.036048 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:26:26.036136 (+ 88us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:26:26.036213 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0806 06:26:26.036412 (+ 199us) server_negotiation.cc:299] Negotiation successful
0806 06:26:26.036474 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
W20260806 06:26:26.036983 22442 tablet_service.cc:2908] Rejecting scan request for tablet 5826563206774a97a436464d671b8955: Uninitialized: clean time has not yet been initialized
I20260806 06:26:26.037914 21554 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHhLhT/build/release/bin/kudu with pid 22389
I20260806 06:26:26.050709 21554 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHhLhT/build/release/bin/kudu with pid 22566
I20260806 06:26:26.061225 21554 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHhLhT/build/release/bin/kudu with pid 22740
I20260806 06:26:26.069046 21554 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHhLhT/build/release/bin/kudu with pid 22267
2026-08-06T06:26:26Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2024 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785997581049705-21554-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 06:26:26 dist-test-slave-4tdj krb5kdc[22970](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 06:26:26 dist-test-slave-4tdj krb5kdc[22970](info): set up 2 sockets
Aug 06 06:26:26 dist-test-slave-4tdj krb5kdc[22970](info): commencing operation
krb5kdc: starting...
W20260806 06:26:27.188834 21554 mini_kdc.cc:121] Time spent starting KDC: real 1.096s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785997581049705-21554-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785997581049705-21554-0/krb5kdc/test-user.keytab.
Aug 06 06:26:27 dist-test-slave-4tdj krb5kdc[22970](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997587, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T06:26:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T06:26:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.12.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.12.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.12.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.190.keytab.
Entry for principal kudu/127.21.12.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.190.keytab.
I20260806 06:26:27.301568 21554 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHhLhT/build/release/bin/kudu
/tmp/dist-test-taskFHhLhT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785997581049705-21554-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785997581049705-21554-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785997581049705-21554-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785997581049705-21554-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.12.190:41067
--webserver_interface=127.21.12.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.190.keytab
--principal=kudu/127.21.12.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.148:37581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.12.190:41067
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785997581049705-21554-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785997581049705-21554-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785997581049705-21554-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 06:26:27.381852 22984 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 06:26:27.382050 22984 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 06:26:27.382079 22984 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 06:26:27.383219 22984 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 06:26:27.383260 22984 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260806 06:26:27.383276 22984 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 06:26:27.383293 22984 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 06:26:27.383308 22984 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 06:26:27.383325 22984 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 06:26:27.384790 22984 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.148:37581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785997581049705-21554-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785997581049705-21554-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.12.190:41067
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.190.keytab
--principal=kudu/127.21.12.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.12.190:41067
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785997581049705-21554-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.12.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22984
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785997581049705-21554-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 05:11:02 UTC on ccycloud.quasar-hvzbbz.root.comops.site
build id 4569617
I20260806 06:26:27.385023 22984 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 06:26:27.385231 22984 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 06:26:27.389796 22990 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:26:27.389791 22993 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:26:27.389791 22991 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:26:27.390076 22984 server_base.cc:1034] running on GCE node
I20260806 06:26:27.390311 22984 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 06:26:27.390607 22984 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 06:26:27.391741 22984 hybrid_clock.cc:648] HybridClock initialized: now 1785997587391730 us; error 30 us; skew 500 ppm
Aug 06 06:26:27 dist-test-slave-4tdj krb5kdc[22970](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997587, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 06:26:27.400164 22984 init.cc:377] Logged in from keytab as kudu/127.21.12.190@KRBTEST.COM (short username kudu)
I20260806 06:26:27.400575 22984 webserver.cc:460] Webserver started at http://127.21.12.190:44345/ using document root <none> and password file <none>
I20260806 06:26:27.400790 22984 fs_manager.cc:359] Metadata directory not provided
I20260806 06:26:27.400835 22984 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 06:26:27.400950 22984 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 06:26:27.401980 22984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785997581049705-21554-0/minicluster-data/master-0/data/instance:
uuid: "31c5e8e609c44afa8e0f2b07d741d0af"
format_stamp: "Formatted at 2026-08-06 06:26:27 on dist-test-slave-4tdj"
server_key: "fdae477fe3ecffadfbb8a75a3c5ba896"
server_key_iv: "e2b055f0b04be6a0d76b3111c5e9d2d2"
server_key_version: "encryptionkey@0"
I20260806 06:26:27.402350 22984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785997581049705-21554-0/minicluster-data/master-0/wal/instance:
uuid: "31c5e8e609c44afa8e0f2b07d741d0af"
format_stamp: "Formatted at 2026-08-06 06:26:27 on dist-test-slave-4tdj"
server_key: "fdae477fe3ecffadfbb8a75a3c5ba896"
server_key_iv: "e2b055f0b04be6a0d76b3111c5e9d2d2"
server_key_version: "encryptionkey@0"
I20260806 06:26:27.403620 22984 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260806 06:26:27.404322 23004 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 06:26:27.404449 22984 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260806 06:26:27.404538 22984 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785997581049705-21554-0/minicluster-data/master-0/data,/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785997581049705-21554-0/minicluster-data/master-0/wal
uuid: "31c5e8e609c44afa8e0f2b07d741d0af"
format_stamp: "Formatted at 2026-08-06 06:26:27 on dist-test-slave-4tdj"
server_key: "fdae477fe3ecffadfbb8a75a3c5ba896"
server_key_iv: "e2b055f0b04be6a0d76b3111c5e9d2d2"
server_key_version: "encryptionkey@0"
I20260806 06:26:27.404599 22984 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785997581049705-21554-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785997581049705-21554-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785997581049705-21554-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 06:26:27.426532 22984 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 06:26:27.428256 22984 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 06:26:27.428386 22984 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 06:26:27.432781 22984 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.190:41067
I20260806 06:26:27.432859 23088 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.190:41067 every 8 connection(s)
I20260806 06:26:27.433152 22984 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785997581049705-21554-0/minicluster-data/master-0/data/info.pb
I20260806 06:26:27.433808 23089 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 06:26:27.436003 23089 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31c5e8e609c44afa8e0f2b07d741d0af: Bootstrap starting.
I20260806 06:26:27.436466 21554 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHhLhT/build/release/bin/kudu as pid 22984
I20260806 06:26:27.436580 21554 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785997581049705-21554-0/minicluster-data/master-0/wal/instance
I20260806 06:26:27.436578 23089 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 31c5e8e609c44afa8e0f2b07d741d0af: Neither blocks nor log segments found. Creating new log.
I20260806 06:26:27.436728 21554 external_mini_cluster.cc:1428] Setting key d7846d55c9c6d587d1928d70167182bc
I20260806 06:26:27.436849 23089 log.cc:824] T 00000000000000000000000000000000 P 31c5e8e609c44afa8e0f2b07d741d0af: Log is configured to *not* fsync() on all Append() calls
I20260806 06:26:27.437644 23089 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31c5e8e609c44afa8e0f2b07d741d0af: No bootstrap required, opened a new log
I20260806 06:26:27.439445 23089 raft_consensus.cc:348] T 00000000000000000000000000000000 P 31c5e8e609c44afa8e0f2b07d741d0af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31c5e8e609c44afa8e0f2b07d741d0af" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 41067 } }
I20260806 06:26:27.439630 23089 raft_consensus.cc:374] T 00000000000000000000000000000000 P 31c5e8e609c44afa8e0f2b07d741d0af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 06:26:27.439666 23089 raft_consensus.cc:729] T 00000000000000000000000000000000 P 31c5e8e609c44afa8e0f2b07d741d0af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31c5e8e609c44afa8e0f2b07d741d0af, State: Initialized, Role: FOLLOWER
I20260806 06:26:27.439790 23089 consensus_queue.cc:260] T 00000000000000000000000000000000 P 31c5e8e609c44afa8e0f2b07d741d0af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31c5e8e609c44afa8e0f2b07d741d0af" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 41067 } }
I20260806 06:26:27.439858 23089 raft_consensus.cc:388] T 00000000000000000000000000000000 P 31c5e8e609c44afa8e0f2b07d741d0af [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 06:26:27.439890 23089 raft_consensus.cc:482] T 00000000000000000000000000000000 P 31c5e8e609c44afa8e0f2b07d741d0af [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 06:26:27.439934 23089 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 31c5e8e609c44afa8e0f2b07d741d0af [term 0 FOLLOWER]: Advancing to term 1
I20260806 06:26:27.440564 23089 raft_consensus.cc:504] T 00000000000000000000000000000000 P 31c5e8e609c44afa8e0f2b07d741d0af [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31c5e8e609c44afa8e0f2b07d741d0af" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 41067 } }
I20260806 06:26:27.440675 23089 leader_election.cc:302] T 00000000000000000000000000000000 P 31c5e8e609c44afa8e0f2b07d741d0af [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 31c5e8e609c44afa8e0f2b07d741d0af; no voters:
I20260806 06:26:27.440837 23089 leader_election.cc:288] T 00000000000000000000000000000000 P 31c5e8e609c44afa8e0f2b07d741d0af [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 06:26:27.440864 23098 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 31c5e8e609c44afa8e0f2b07d741d0af [term 1 FOLLOWER]: Leader election won for term 1
I20260806 06:26:27.440977 23098 raft_consensus.cc:686] T 00000000000000000000000000000000 P 31c5e8e609c44afa8e0f2b07d741d0af [term 1 LEADER]: Becoming Leader. State: Replica: 31c5e8e609c44afa8e0f2b07d741d0af, State: Running, Role: LEADER
I20260806 06:26:27.441066 23098 consensus_queue.cc:237] T 00000000000000000000000000000000 P 31c5e8e609c44afa8e0f2b07d741d0af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31c5e8e609c44afa8e0f2b07d741d0af" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 41067 } }
I20260806 06:26:27.441090 23089 sys_catalog.cc:564] T 00000000000000000000000000000000 P 31c5e8e609c44afa8e0f2b07d741d0af [sys.catalog]: configured and running, proceeding with master startup.
I20260806 06:26:27.441628 23099 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31c5e8e609c44afa8e0f2b07d741d0af [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "31c5e8e609c44afa8e0f2b07d741d0af" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31c5e8e609c44afa8e0f2b07d741d0af" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 41067 } } }
I20260806 06:26:27.441779 23099 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31c5e8e609c44afa8e0f2b07d741d0af [sys.catalog]: This master's current role is: LEADER
I20260806 06:26:27.442241 23107 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 06:26:27.442716 23107 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 06:26:27.442826 23100 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31c5e8e609c44afa8e0f2b07d741d0af [sys.catalog]: SysCatalogTable state changed. Reason: New leader 31c5e8e609c44afa8e0f2b07d741d0af. Latest consensus state: current_term: 1 leader_uuid: "31c5e8e609c44afa8e0f2b07d741d0af" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31c5e8e609c44afa8e0f2b07d741d0af" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 41067 } } }
I20260806 06:26:27.443957 23100 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31c5e8e609c44afa8e0f2b07d741d0af [sys.catalog]: This master's current role is: LEADER
Aug 06 06:26:27 dist-test-slave-4tdj krb5kdc[22970](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997587, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.12.190@KRBTEST.COM
I20260806 06:26:27.445374 23107 catalog_manager.cc:1347] Generated new cluster ID: b60c8e4565d343dd98ac498ad6a6d679
I20260806 06:26:27.445423 23107 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 06:26:27.448297 23096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:27.437192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48742 (local address 127.21.12.190:41067)
0806 06:26:27.437348 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0806 06:26:27.437358 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 06:26:27.437391 (+ 33us) server_negotiation.cc:407] Connection header received
0806 06:26:27.437925 (+ 534us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:26:27.437934 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:26:27.437997 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:26:27.438153 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:26:27.438733 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:27.440494 (+ 1761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:27.441454 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:27.442096 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:27.442209 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:27.445017 (+ 2808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:26:27.445021 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:26:27.445021 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 06:26:27.445030 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:26:27.446636 (+ 1606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:26:27.446949 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:26:27.446953 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:26:27.446953 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:26:27.447030 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:26:27.447199 (+ 169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:26:27.447200 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:26:27.447200 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:26:27.447411 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:26:27.447999 (+ 588us) server_negotiation.cc:1035] Waiting for connection context
0806 06:26:27.448058 (+ 59us) server_negotiation.cc:299] Negotiation successful
0806 06:26:27.448159 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":67,"threads_started":1}
I20260806 06:26:27.472678 23107 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 06:26:27.473201 23107 catalog_manager.cc:1504] Loading token signing keys...
I20260806 06:26:27.476672 23107 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 31c5e8e609c44afa8e0f2b07d741d0af: Generated new TSK 0
I20260806 06:26:27.476826 23107 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.12.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.12.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.12.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.129.keytab.
Entry for principal kudu/127.21.12.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.129.keytab.
I20260806 06:26:27.504493 21554 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHhLhT/build/release/bin/kudu
/tmp/dist-test-taskFHhLhT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785997581049705-21554-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785997581049705-21554-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785997581049705-21554-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785997581049705-21554-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.12.129:0
--local_ip_for_outbound_sockets=127.21.12.129
--webserver_interface=127.21.12.129
--webserver_port=0
--tserver_master_addrs=127.21.12.190:41067
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.129.keytab
--principal=kudu/127.21.12.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.148:37581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785997581049705-21554-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785997581049705-21554-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785997581049705-21554-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 06:26:27.592074 23124 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 06:26:27.592294 23124 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 06:26:27.592316 23124 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 06:26:27.593394 23124 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 06:26:27.593426 23124 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 06:26:27.593446 23124 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 06:26:27.593466 23124 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.12.129
I20260806 06:26:27.594892 23124 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.148:37581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785997581049705-21554-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785997581049705-21554-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.129.keytab
--principal=kudu/127.21.12.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.12.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785997581049705-21554-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.12.129
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.21.12.190:41067
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23124
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.12.129
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785997581049705-21554-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 05:11:02 UTC on ccycloud.quasar-hvzbbz.root.comops.site
build id 4569617
I20260806 06:26:27.595068 23124 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 06:26:27.595243 23124 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 06:26:27.599948 23130 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:26:27.599973 23131 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:26:27.599972 23124 server_base.cc:1034] running on GCE node
W20260806 06:26:27.599956 23133 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:26:27.600291 23124 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 06:26:27.600513 23124 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 06:26:27.601676 23124 hybrid_clock.cc:648] HybridClock initialized: now 1785997587601658 us; error 32 us; skew 500 ppm
Aug 06 06:26:27 dist-test-slave-4tdj krb5kdc[22970](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997587, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 06:26:27.604004 23124 init.cc:377] Logged in from keytab as kudu/127.21.12.129@KRBTEST.COM (short username kudu)
I20260806 06:26:27.604370 23124 webserver.cc:460] Webserver started at http://127.21.12.129:35357/ using document root <none> and password file <none>
I20260806 06:26:27.604539 23124 fs_manager.cc:359] Metadata directory not provided
I20260806 06:26:27.604578 23124 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 06:26:27.604665 23124 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 06:26:27.605633 23124 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785997581049705-21554-0/minicluster-data/ts-0/data/instance:
uuid: "c4cd0bf3767a4024ba327faf3015304c"
format_stamp: "Formatted at 2026-08-06 06:26:27 on dist-test-slave-4tdj"
server_key: "33a685e92682fd616659bc542d2ff2e6"
server_key_iv: "2d10ff4b7b633b94981c930161f9100b"
server_key_version: "encryptionkey@0"
I20260806 06:26:27.605916 23124 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785997581049705-21554-0/minicluster-data/ts-0/wal/instance:
uuid: "c4cd0bf3767a4024ba327faf3015304c"
format_stamp: "Formatted at 2026-08-06 06:26:27 on dist-test-slave-4tdj"
server_key: "33a685e92682fd616659bc542d2ff2e6"
server_key_iv: "2d10ff4b7b633b94981c930161f9100b"
server_key_version: "encryptionkey@0"
I20260806 06:26:27.607240 23124 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260806 06:26:27.607910 23142 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 06:26:27.608150 23124 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260806 06:26:27.608265 23124 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785997581049705-21554-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785997581049705-21554-0/minicluster-data/ts-0/wal
uuid: "c4cd0bf3767a4024ba327faf3015304c"
format_stamp: "Formatted at 2026-08-06 06:26:27 on dist-test-slave-4tdj"
server_key: "33a685e92682fd616659bc542d2ff2e6"
server_key_iv: "2d10ff4b7b633b94981c930161f9100b"
server_key_version: "encryptionkey@0"
I20260806 06:26:27.608355 23124 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785997581049705-21554-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785997581049705-21554-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785997581049705-21554-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 06:26:27.623363 23124 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 06:26:27.625241 23124 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 06:26:27.625360 23124 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 06:26:27.625836 23124 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 06:26:27.625871 23124 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 06:26:27.625892 23124 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 06:26:27.625905 23124 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 06:26:27.632774 23124 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.129:33499
I20260806 06:26:27.632872 23303 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.129:33499 every 8 connection(s)
Aug 06 06:26:27 dist-test-slave-4tdj krb5kdc[22970](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997587, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.129@KRBTEST.COM for kudu/127.21.12.190@KRBTEST.COM
I20260806 06:26:27.633145 23124 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785997581049705-21554-0/minicluster-data/ts-0/data/info.pb
I20260806 06:26:27.635700 23096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:27.626420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.129:43363 (local address 127.21.12.190:41067)
0806 06:26:27.626489 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0806 06:26:27.626492 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 06:26:27.626730 (+ 238us) server_negotiation.cc:407] Connection header received
0806 06:26:27.627372 (+ 642us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:26:27.627374 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:26:27.627385 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:26:27.627445 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:26:27.628246 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:27.629163 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:27.630667 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:27.630991 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:27.631054 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:27.633591 (+ 2537us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:26:27.633593 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:26:27.633593 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 06:26:27.633600 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:26:27.634718 (+ 1118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:26:27.635048 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:26:27.635048 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:26:27.635049 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:26:27.635090 (+ 41us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:26:27.635259 (+ 169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:26:27.635259 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:26:27.635259 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:26:27.635346 (+ 87us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:26:27.635388 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0806 06:26:27.635576 (+ 188us) server_negotiation.cc:299] Negotiation successful
0806 06:26:27.635637 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260806 06:26:27.635821 23154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:27.626402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:41067 (local address 127.21.12.129:43363)
0806 06:26:27.626595 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0806 06:26:27.626645 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0806 06:26:27.627269 (+ 624us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:26:27.627523 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:26:27.627528 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:26:27.627876 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0806 06:26:27.628161 (+ 285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:27.628168 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:27.629278 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:27.629280 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:27.630587 (+ 1307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:27.630591 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:27.631075 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:27.631077 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:27.631218 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:27.631750 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 06:26:27.631760 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0806 06:26:27.633457 (+ 1697us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 06:26:27.634793 (+ 1336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:26:27.634800 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:26:27.634802 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:26:27.634996 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:26:27.635132 (+ 136us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:26:27.635133 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:26:27.635133 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:26:27.635217 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:26:27.635384 (+ 167us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 06:26:27.635387 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 06:26:27.635483 (+ 96us) client_negotiation.cc:769] Sending connection context
0806 06:26:27.635562 (+ 79us) client_negotiation.cc:240] Negotiation successful
0806 06:26:27.635668 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":100,"threads_started":1}
I20260806 06:26:27.636269 23305 heartbeater.cc:347] Connected to a master server at 127.21.12.190:41067
I20260806 06:26:27.636345 23305 heartbeater.cc:464] Registering TS with master...
I20260806 06:26:27.636497 23305 heartbeater.cc:511] Master 127.21.12.190:41067 requested a full tablet report, sending...
I20260806 06:26:27.637024 23030 ts_manager.cc:194] Registered new tserver with Master: c4cd0bf3767a4024ba327faf3015304c (127.21.12.129:33499)
I20260806 06:26:27.637770 23030 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.12.129@KRBTEST.COM'} at 127.21.12.129:43363
I20260806 06:26:27.639804 21554 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHhLhT/build/release/bin/kudu as pid 23124
I20260806 06:26:27.639896 21554 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785997581049705-21554-0/minicluster-data/ts-0/wal/instance
I20260806 06:26:27.640046 21554 external_mini_cluster.cc:1428] Setting key 198cafc30ca8d74b4c73967e0705d8cc
I20260806 06:26:27.642189 23096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:27.637284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.129:57219 (local address 127.21.12.190:41067)
0806 06:26:27.637323 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0806 06:26:27.637325 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:26:27.637434 (+ 109us) server_negotiation.cc:407] Connection header received
0806 06:26:27.637458 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:26:27.637459 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:26:27.637468 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:26:27.637520 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:26:27.637785 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:27.638271 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:27.638961 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:27.639132 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:27.639174 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:27.639920 (+ 746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:26:27.639921 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:26:27.639922 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 06:26:27.639928 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:26:27.641202 (+ 1274us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:26:27.641596 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:26:27.641597 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:26:27.641597 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:26:27.641646 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:26:27.641822 (+ 176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:26:27.641822 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:26:27.641823 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:26:27.641907 (+ 84us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:26:27.641952 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0806 06:26:27.642081 (+ 129us) server_negotiation.cc:299] Negotiation successful
0806 06:26:27.642136 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260806 06:26:27.642246 23312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:27.637227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:41067 (local address 127.21.12.129:57219)
0806 06:26:27.637374 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0806 06:26:27.637382 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0806 06:26:27.637429 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:26:27.637544 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:26:27.637545 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:26:27.637683 (+ 138us) client_negotiation.cc:189] Negotiated authn=SASL
0806 06:26:27.637731 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:27.637732 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:27.638362 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:27.638363 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:27.638896 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:27.638898 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:27.639181 (+ 283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:27.639182 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:27.639236 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:27.639543 (+ 307us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 06:26:27.639547 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0806 06:26:27.639825 (+ 278us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 06:26:27.641276 (+ 1451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:26:27.641277 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:26:27.641277 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:26:27.641477 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:26:27.641716 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:26:27.641716 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:26:27.641717 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:26:27.641776 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:26:27.641954 (+ 178us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 06:26:27.641955 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 06:26:27.642007 (+ 52us) client_negotiation.cc:769] Sending connection context
0806 06:26:27.642065 (+ 58us) client_negotiation.cc:240] Negotiation successful
0806 06:26:27.642154 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":75,"threads_started":1}
WARNING: no policy specified for kudu/127.21.12.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.12.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.12.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.130.keytab.
Entry for principal kudu/127.21.12.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.130.keytab.
I20260806 06:26:27.663237 21554 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHhLhT/build/release/bin/kudu
/tmp/dist-test-taskFHhLhT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785997581049705-21554-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785997581049705-21554-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785997581049705-21554-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785997581049705-21554-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.12.130:0
--local_ip_for_outbound_sockets=127.21.12.130
--webserver_interface=127.21.12.130
--webserver_port=0
--tserver_master_addrs=127.21.12.190:41067
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.130.keytab
--principal=kudu/127.21.12.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.148:37581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785997581049705-21554-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785997581049705-21554-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785997581049705-21554-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 06:26:27.743862 23317 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 06:26:27.744051 23317 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 06:26:27.744072 23317 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 06:26:27.745193 23317 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 06:26:27.745244 23317 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 06:26:27.745263 23317 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 06:26:27.745280 23317 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.12.130
I20260806 06:26:27.746865 23317 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.148:37581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785997581049705-21554-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785997581049705-21554-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.130.keytab
--principal=kudu/127.21.12.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.12.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785997581049705-21554-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.12.130
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.21.12.190:41067
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23317
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.12.130
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785997581049705-21554-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 05:11:02 UTC on ccycloud.quasar-hvzbbz.root.comops.site
build id 4569617
I20260806 06:26:27.747104 23317 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 06:26:27.747334 23317 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 06:26:27.752372 23322 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:26:27.752405 23323 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:26:27.752410 23317 server_base.cc:1034] running on GCE node
W20260806 06:26:27.752393 23326 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:26:27.752760 23317 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 06:26:27.752988 23317 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 06:26:27.754149 23317 hybrid_clock.cc:648] HybridClock initialized: now 1785997587754134 us; error 27 us; skew 500 ppm
Aug 06 06:26:27 dist-test-slave-4tdj krb5kdc[22970](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997587, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 06:26:27.756642 23317 init.cc:377] Logged in from keytab as kudu/127.21.12.130@KRBTEST.COM (short username kudu)
I20260806 06:26:27.757042 23317 webserver.cc:460] Webserver started at http://127.21.12.130:37177/ using document root <none> and password file <none>
I20260806 06:26:27.757246 23317 fs_manager.cc:359] Metadata directory not provided
I20260806 06:26:27.757293 23317 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 06:26:27.757429 23317 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 06:26:27.758512 23317 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785997581049705-21554-0/minicluster-data/ts-1/data/instance:
uuid: "bdd4da71739a455c9495a8887b22fa09"
format_stamp: "Formatted at 2026-08-06 06:26:27 on dist-test-slave-4tdj"
server_key: "46b94d17a2e664a9c798712a6887dec4"
server_key_iv: "549c17e8dfba83966ea16bddae5aeda9"
server_key_version: "encryptionkey@0"
I20260806 06:26:27.758845 23317 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785997581049705-21554-0/minicluster-data/ts-1/wal/instance:
uuid: "bdd4da71739a455c9495a8887b22fa09"
format_stamp: "Formatted at 2026-08-06 06:26:27 on dist-test-slave-4tdj"
server_key: "46b94d17a2e664a9c798712a6887dec4"
server_key_iv: "549c17e8dfba83966ea16bddae5aeda9"
server_key_version: "encryptionkey@0"
I20260806 06:26:27.760070 23317 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260806 06:26:27.760710 23334 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 06:26:27.760866 23317 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260806 06:26:27.760960 23317 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785997581049705-21554-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785997581049705-21554-0/minicluster-data/ts-1/wal
uuid: "bdd4da71739a455c9495a8887b22fa09"
format_stamp: "Formatted at 2026-08-06 06:26:27 on dist-test-slave-4tdj"
server_key: "46b94d17a2e664a9c798712a6887dec4"
server_key_iv: "549c17e8dfba83966ea16bddae5aeda9"
server_key_version: "encryptionkey@0"
I20260806 06:26:27.761020 23317 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785997581049705-21554-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785997581049705-21554-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785997581049705-21554-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 06:26:27.770062 23317 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 06:26:27.771792 23317 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 06:26:27.771916 23317 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 06:26:27.772354 23317 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 06:26:27.772392 23317 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 06:26:27.772420 23317 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 06:26:27.772437 23317 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 06:26:27 dist-test-slave-4tdj krb5kdc[22970](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997587, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.130@KRBTEST.COM for kudu/127.21.12.190@KRBTEST.COM
I20260806 06:26:27.779311 23317 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.130:33755
I20260806 06:26:27.779633 23317 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785997581049705-21554-0/minicluster-data/ts-1/data/info.pb
I20260806 06:26:27.779641 23505 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.130:33755 every 8 connection(s)
I20260806 06:26:27.781481 23096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:27.772764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.130:37377 (local address 127.21.12.190:41067)
0806 06:26:27.772827 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0806 06:26:27.772830 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 06:26:27.773015 (+ 185us) server_negotiation.cc:407] Connection header received
0806 06:26:27.773647 (+ 632us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:26:27.773649 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:26:27.773662 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:26:27.773726 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:26:27.774567 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:27.775110 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:27.776075 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:27.776286 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:27.776342 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:27.778898 (+ 2556us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:26:27.778900 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:26:27.778900 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 06:26:27.778908 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:26:27.780251 (+ 1343us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:26:27.780631 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:26:27.780633 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:26:27.780633 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:26:27.780689 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:26:27.780896 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:26:27.780898 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:26:27.780898 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:26:27.781007 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:26:27.781070 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0806 06:26:27.781315 (+ 245us) server_negotiation.cc:299] Negotiation successful
0806 06:26:27.781390 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260806 06:26:27.781632 23346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:27.772730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:41067 (local address 127.21.12.130:37377)
0806 06:26:27.772879 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0806 06:26:27.772925 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0806 06:26:27.773530 (+ 605us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:26:27.773811 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:26:27.773815 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:26:27.774177 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0806 06:26:27.774471 (+ 294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:27.774478 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:27.775200 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:27.775201 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:27.775993 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:27.775996 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:27.776365 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:27.776366 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:27.776491 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:27.776919 (+ 428us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 06:26:27.776928 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0806 06:26:27.778813 (+ 1885us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 06:26:27.780321 (+ 1508us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:26:27.780328 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:26:27.780330 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:26:27.780566 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:26:27.780741 (+ 175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:26:27.780742 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:26:27.780742 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:26:27.780843 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:26:27.781063 (+ 220us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 06:26:27.781069 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 06:26:27.781201 (+ 132us) client_negotiation.cc:769] Sending connection context
0806 06:26:27.781285 (+ 84us) client_negotiation.cc:240] Negotiation successful
0806 06:26:27.781428 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":68,"threads_started":1}
I20260806 06:26:27.782047 23506 heartbeater.cc:347] Connected to a master server at 127.21.12.190:41067
I20260806 06:26:27.782136 23506 heartbeater.cc:464] Registering TS with master...
I20260806 06:26:27.782328 23506 heartbeater.cc:511] Master 127.21.12.190:41067 requested a full tablet report, sending...
I20260806 06:26:27.782703 23031 ts_manager.cc:194] Registered new tserver with Master: bdd4da71739a455c9495a8887b22fa09 (127.21.12.130:33755)
I20260806 06:26:27.783171 23031 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.12.130@KRBTEST.COM'} at 127.21.12.130:37377
I20260806 06:26:27.787611 23096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:27.783073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.130:55157 (local address 127.21.12.190:41067)
0806 06:26:27.783108 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0806 06:26:27.783109 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0806 06:26:27.783200 (+ 91us) server_negotiation.cc:407] Connection header received
0806 06:26:27.783256 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:26:27.783257 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:26:27.783266 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:26:27.783317 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:26:27.783657 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:27.784092 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:27.784713 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:27.784875 (+ 162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:27.784916 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:27.785665 (+ 749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:26:27.785665 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:26:27.785666 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 06:26:27.785670 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:26:27.786687 (+ 1017us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:26:27.786951 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:26:27.786951 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:26:27.786952 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:26:27.786993 (+ 41us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:26:27.787122 (+ 129us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:26:27.787122 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:26:27.787122 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:26:27.787235 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:26:27.787286 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0806 06:26:27.787505 (+ 219us) server_negotiation.cc:299] Negotiation successful
0806 06:26:27.787558 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260806 06:26:27.787623 23514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:27.783023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:41067 (local address 127.21.12.130:55157)
0806 06:26:27.783149 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0806 06:26:27.783159 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0806 06:26:27.783212 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:26:27.783337 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:26:27.783338 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:26:27.783509 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0806 06:26:27.783574 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:27.783576 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:27.784174 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:27.784175 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:27.784656 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:27.784657 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:27.784921 (+ 264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:27.784922 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:27.784973 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:27.785338 (+ 365us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 06:26:27.785342 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0806 06:26:27.785607 (+ 265us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 06:26:27.786753 (+ 1146us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:26:27.786754 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:26:27.786755 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:26:27.786898 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:26:27.787045 (+ 147us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:26:27.787046 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:26:27.787046 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:26:27.787089 (+ 43us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:26:27.787361 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 06:26:27.787362 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 06:26:27.787404 (+ 42us) client_negotiation.cc:769] Sending connection context
0806 06:26:27.787484 (+ 80us) client_negotiation.cc:240] Negotiation successful
0806 06:26:27.787560 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":68,"threads_started":1}
I20260806 06:26:27.787842 21554 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHhLhT/build/release/bin/kudu as pid 23317
I20260806 06:26:27.787922 21554 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785997581049705-21554-0/minicluster-data/ts-1/wal/instance
I20260806 06:26:27.788058 21554 external_mini_cluster.cc:1428] Setting key 6c93673d88cc4e83edb25b0042adf4ee
WARNING: no policy specified for kudu/127.21.12.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.12.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.12.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.131.keytab.
Entry for principal kudu/127.21.12.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.131.keytab.
I20260806 06:26:27.810595 21554 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHhLhT/build/release/bin/kudu
/tmp/dist-test-taskFHhLhT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785997581049705-21554-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785997581049705-21554-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785997581049705-21554-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785997581049705-21554-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.12.131:0
--local_ip_for_outbound_sockets=127.21.12.131
--webserver_interface=127.21.12.131
--webserver_port=0
--tserver_master_addrs=127.21.12.190:41067
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.131.keytab
--principal=kudu/127.21.12.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.148:37581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785997581049705-21554-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785997581049705-21554-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785997581049705-21554-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 06:26:27.890600 23519 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 06:26:27.890797 23519 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 06:26:27.890827 23519 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 06:26:27.891996 23519 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 06:26:27.892033 23519 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 06:26:27.892060 23519 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 06:26:27.892082 23519 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.12.131
I20260806 06:26:27.893483 23519 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.148:37581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785997581049705-21554-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785997581049705-21554-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.131.keytab
--principal=kudu/127.21.12.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.12.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785997581049705-21554-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.12.131
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.21.12.190:41067
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23519
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.12.131
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785997581049705-21554-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 05:11:02 UTC on ccycloud.quasar-hvzbbz.root.comops.site
build id 4569617
I20260806 06:26:27.893724 23519 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 06:26:27.893929 23519 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 06:26:27.898761 23524 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:26:27.898752 23530 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:26:27.898751 23526 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:26:27.899101 23519 server_base.cc:1034] running on GCE node
I20260806 06:26:27.899267 23519 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 06:26:27.899477 23519 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 06:26:27.900617 23519 hybrid_clock.cc:648] HybridClock initialized: now 1785997587900601 us; error 35 us; skew 500 ppm
Aug 06 06:26:27 dist-test-slave-4tdj krb5kdc[22970](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997587, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 06:26:27.903015 23519 init.cc:377] Logged in from keytab as kudu/127.21.12.131@KRBTEST.COM (short username kudu)
I20260806 06:26:27.903399 23519 webserver.cc:460] Webserver started at http://127.21.12.131:40533/ using document root <none> and password file <none>
I20260806 06:26:27.903585 23519 fs_manager.cc:359] Metadata directory not provided
I20260806 06:26:27.903622 23519 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 06:26:27.903716 23519 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 06:26:27.904713 23519 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785997581049705-21554-0/minicluster-data/ts-2/data/instance:
uuid: "279a4037d58b470d976dc6f43aff7790"
format_stamp: "Formatted at 2026-08-06 06:26:27 on dist-test-slave-4tdj"
server_key: "33a3c218f728e2783752bb84cb80db35"
server_key_iv: "24562f7a29ce46ef69ce6ac785f811cc"
server_key_version: "encryptionkey@0"
I20260806 06:26:27.905004 23519 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785997581049705-21554-0/minicluster-data/ts-2/wal/instance:
uuid: "279a4037d58b470d976dc6f43aff7790"
format_stamp: "Formatted at 2026-08-06 06:26:27 on dist-test-slave-4tdj"
server_key: "33a3c218f728e2783752bb84cb80db35"
server_key_iv: "24562f7a29ce46ef69ce6ac785f811cc"
server_key_version: "encryptionkey@0"
I20260806 06:26:27.906275 23519 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260806 06:26:27.907068 23538 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 06:26:27.907220 23519 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260806 06:26:27.907311 23519 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785997581049705-21554-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785997581049705-21554-0/minicluster-data/ts-2/wal
uuid: "279a4037d58b470d976dc6f43aff7790"
format_stamp: "Formatted at 2026-08-06 06:26:27 on dist-test-slave-4tdj"
server_key: "33a3c218f728e2783752bb84cb80db35"
server_key_iv: "24562f7a29ce46ef69ce6ac785f811cc"
server_key_version: "encryptionkey@0"
I20260806 06:26:27.907372 23519 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785997581049705-21554-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785997581049705-21554-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785997581049705-21554-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 06:26:27.923300 23519 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 06:26:27.925577 23519 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 06:26:27.925726 23519 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 06:26:27.926327 23519 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 06:26:27.926391 23519 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 06:26:27.926427 23519 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 06:26:27.926466 23519 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 06:26:27 dist-test-slave-4tdj krb5kdc[22970](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997587, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.131@KRBTEST.COM for kudu/127.21.12.190@KRBTEST.COM
I20260806 06:26:27.934115 23519 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.131:39473
I20260806 06:26:27.934249 23708 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.131:39473 every 8 connection(s)
I20260806 06:26:27.934573 23519 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785997581049705-21554-0/minicluster-data/ts-2/data/info.pb
I20260806 06:26:27.935192 21554 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHhLhT/build/release/bin/kudu as pid 23519
I20260806 06:26:27.935284 21554 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785997581049705-21554-0/minicluster-data/ts-2/wal/instance
I20260806 06:26:27.935423 21554 external_mini_cluster.cc:1428] Setting key 1989e832dd02c8521d7891aee1aaf11f
I20260806 06:26:27.935453 23096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:27.926740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.131:38189 (local address 127.21.12.190:41067)
0806 06:26:27.926793 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0806 06:26:27.926795 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:26:27.927006 (+ 211us) server_negotiation.cc:407] Connection header received
0806 06:26:27.927500 (+ 494us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:26:27.927501 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:26:27.927517 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:26:27.927603 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:26:27.928185 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:27.928689 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:27.929784 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:27.929979 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:27.930034 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:27.932410 (+ 2376us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:26:27.932412 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:26:27.932412 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 06:26:27.932419 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:26:27.933805 (+ 1386us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:26:27.934409 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:26:27.934411 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:26:27.934412 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:26:27.934475 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:26:27.934767 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:26:27.934768 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:26:27.934768 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:26:27.934886 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:26:27.934950 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0806 06:26:27.935282 (+ 332us) server_negotiation.cc:299] Negotiation successful
0806 06:26:27.935363 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260806 06:26:27.935591 23552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:27.926739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:41067 (local address 127.21.12.131:38189)
0806 06:26:27.926885 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0806 06:26:27.926919 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0806 06:26:27.927394 (+ 475us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:26:27.927655 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:26:27.927659 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:26:27.927878 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0806 06:26:27.928102 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:27.928108 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:27.928775 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:27.928776 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:27.929705 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:27.929708 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:27.930054 (+ 346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:27.930055 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:27.930169 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:27.930683 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 06:26:27.930692 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0806 06:26:27.932313 (+ 1621us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 06:26:27.933980 (+ 1667us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:26:27.933987 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:26:27.933990 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:26:27.934323 (+ 333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:26:27.934544 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:26:27.934545 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:26:27.934545 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:26:27.934681 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:26:27.934977 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 06:26:27.934984 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 06:26:27.935142 (+ 158us) client_negotiation.cc:769] Sending connection context
0806 06:26:27.935243 (+ 101us) client_negotiation.cc:240] Negotiation successful
0806 06:26:27.935390 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":84,"threads_started":1}
I20260806 06:26:27.936318 23711 heartbeater.cc:347] Connected to a master server at 127.21.12.190:41067
I20260806 06:26:27.936436 23711 heartbeater.cc:464] Registering TS with master...
I20260806 06:26:27.936662 23711 heartbeater.cc:511] Master 127.21.12.190:41067 requested a full tablet report, sending...
I20260806 06:26:27.937270 23031 ts_manager.cc:194] Registered new tserver with Master: 279a4037d58b470d976dc6f43aff7790 (127.21.12.131:39473)
I20260806 06:26:27.937729 23031 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.12.131@KRBTEST.COM'} at 127.21.12.131:38189
I20260806 06:26:27.942730 23096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:27.937858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.131:34051 (local address 127.21.12.190:41067)
0806 06:26:27.937908 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0806 06:26:27.937910 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:26:27.937917 (+ 7us) server_negotiation.cc:407] Connection header received
0806 06:26:27.937935 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:26:27.937936 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:26:27.937945 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:26:27.937993 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:26:27.938372 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:27.938838 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:27.939480 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:27.939684 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:27.939734 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:27.940501 (+ 767us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:26:27.940502 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:26:27.940527 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 06:26:27.940532 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:26:27.941805 (+ 1273us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:26:27.942124 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:26:27.942125 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:26:27.942125 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:26:27.942169 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:26:27.942369 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:26:27.942370 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:26:27.942370 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:26:27.942461 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:26:27.942537 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0806 06:26:27.942621 (+ 84us) server_negotiation.cc:299] Negotiation successful
0806 06:26:27.942673 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260806 06:26:27.942754 23718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:27.937291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:41067 (local address 127.21.12.131:34051)
0806 06:26:27.937434 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0806 06:26:27.937445 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0806 06:26:27.937528 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:26:27.938013 (+ 485us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:26:27.938014 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:26:27.938210 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0806 06:26:27.938272 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:27.938274 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:27.938925 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:27.938926 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:27.939408 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:27.939410 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:27.939738 (+ 328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:27.939738 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:27.939787 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:27.940163 (+ 376us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 06:26:27.940167 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0806 06:26:27.940439 (+ 272us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 06:26:27.941881 (+ 1442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:26:27.941882 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:26:27.941882 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:26:27.942062 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:26:27.942216 (+ 154us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:26:27.942217 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:26:27.942217 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:26:27.942283 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:26:27.942508 (+ 225us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 06:26:27.942509 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 06:26:27.942546 (+ 37us) client_negotiation.cc:769] Sending connection context
0806 06:26:27.942603 (+ 57us) client_negotiation.cc:240] Negotiation successful
0806 06:26:27.942682 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":92,"threads_started":1}
I20260806 06:26:27.947105 21554 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 06 06:26:27 dist-test-slave-4tdj krb5kdc[22970](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997587, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 06 06:26:27 dist-test-slave-4tdj krb5kdc[22970](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997587, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.21.12.190@KRBTEST.COM
I20260806 06:26:27.968883 23096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:27.962942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33586 (local address 127.21.12.190:41067)
0806 06:26:27.962998 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0806 06:26:27.963000 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:26:27.963007 (+ 7us) server_negotiation.cc:407] Connection header received
0806 06:26:27.963030 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:26:27.963031 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:26:27.963045 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:26:27.963092 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:26:27.963615 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:27.964116 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:27.964827 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:27.965005 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:27.965056 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:27.967037 (+ 1981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:26:27.967044 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:26:27.967044 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 06:26:27.967051 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:26:27.968090 (+ 1039us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:26:27.968369 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:26:27.968369 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:26:27.968370 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:26:27.968409 (+ 39us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:26:27.968535 (+ 126us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:26:27.968535 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:26:27.968535 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:26:27.968628 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:26:27.968694 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0806 06:26:27.968764 (+ 70us) server_negotiation.cc:299] Negotiation successful
0806 06:26:27.968816 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260806 06:26:27.970404 23031 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33586:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 06:26:27.971932 23031 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 06:26:27.977978 23730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:27.973921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.130:33755 (local address 127.0.0.1:45980)
0806 06:26:27.974265 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0806 06:26:27.974276 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0806 06:26:27.974483 (+ 207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:26:27.974645 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:26:27.974648 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:26:27.974650 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 06:26:27.974715 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:27.974717 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:27.975962 (+ 1245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:27.975963 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:27.976995 (+ 1032us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:27.976997 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:27.977649 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:27.977650 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:27.977754 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:27.977770 (+ 16us) client_negotiation.cc:769] Sending connection context
0806 06:26:27.977835 (+ 65us) client_negotiation.cc:240] Negotiation successful
0806 06:26:27.977885 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":55,"threads_started":1}
I20260806 06:26:27.978117 23728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:27.974107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.131:39473 (local address 127.0.0.1:56918)
0806 06:26:27.974400 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0806 06:26:27.974408 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0806 06:26:27.974446 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:26:27.975017 (+ 571us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:26:27.975018 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:26:27.975020 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 06:26:27.975056 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:27.975058 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:27.976229 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:27.976230 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:27.977066 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:27.977068 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:27.977921 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:27.977922 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:27.977977 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:27.977981 (+ 4us) client_negotiation.cc:769] Sending connection context
0806 06:26:27.978010 (+ 29us) client_negotiation.cc:240] Negotiation successful
0806 06:26:27.978058 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":32,"threads_started":1}
I20260806 06:26:27.978169 23733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:27.974102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45980 (local address 127.21.12.130:33755)
0806 06:26:27.974472 (+ 370us) server_negotiation.cc:206] Beginning negotiation
0806 06:26:27.974477 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 06:26:27.974485 (+ 8us) server_negotiation.cc:407] Connection header received
0806 06:26:27.974538 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:26:27.974539 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:26:27.974552 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:26:27.974613 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 06:26:27.974800 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:27.975880 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:27.977077 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:27.977571 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:27.977637 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:27.977696 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0806 06:26:27.978033 (+ 337us) server_negotiation.cc:299] Negotiation successful
0806 06:26:27.978075 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":336,"thread_start_us":78,"threads_started":1}
I20260806 06:26:27.977978 23731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:27.973917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.129:33499 (local address 127.0.0.1:40312)
0806 06:26:27.974206 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0806 06:26:27.974248 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0806 06:26:27.974332 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:26:27.974748 (+ 416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:26:27.974749 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:26:27.974751 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 06:26:27.974812 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:27.974814 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:27.975678 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:27.975679 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:27.976897 (+ 1218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:27.976900 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:27.977786 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:27.977787 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:27.977841 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:27.977844 (+ 3us) client_negotiation.cc:769] Sending connection context
0806 06:26:27.977889 (+ 45us) client_negotiation.cc:240] Negotiation successful
0806 06:26:27.977920 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":62,"threads_started":1}
I20260806 06:26:27.978466 23734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:27.974219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56918 (local address 127.21.12.131:39473)
0806 06:26:27.974864 (+ 645us) server_negotiation.cc:206] Beginning negotiation
0806 06:26:27.974867 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 06:26:27.974876 (+ 9us) server_negotiation.cc:407] Connection header received
0806 06:26:27.974909 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:26:27.974911 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:26:27.974924 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:26:27.975124 (+ 200us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 06:26:27.975181 (+ 57us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:27.976162 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:27.977153 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:27.977834 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:27.978215 (+ 381us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:27.978280 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0806 06:26:27.978337 (+ 57us) server_negotiation.cc:299] Negotiation successful
0806 06:26:27.978374 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":608,"thread_start_us":119,"threads_started":1}
I20260806 06:26:27.978890 23411 tablet_service.cc:1467] Processing CreateTablet for tablet 56f29a9251e748419c5a8e3941ae9104 (DEFAULT_TABLE table=test-table [id=7a300369a9244afbb12128bfdd868f22]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 06:26:27.979249 23411 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56f29a9251e748419c5a8e3941ae9104. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 06:26:27.981881 23737 tablet_bootstrap.cc:492] T 56f29a9251e748419c5a8e3941ae9104 P bdd4da71739a455c9495a8887b22fa09: Bootstrap starting.
I20260806 06:26:27.978037 23735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:27.974225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40312 (local address 127.21.12.129:33499)
0806 06:26:27.974588 (+ 363us) server_negotiation.cc:206] Beginning negotiation
0806 06:26:27.974592 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 06:26:27.974601 (+ 9us) server_negotiation.cc:407] Connection header received
0806 06:26:27.974640 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:26:27.974642 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:26:27.974657 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:26:27.974735 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 06:26:27.974895 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:27.975602 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:27.976975 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:27.977717 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:27.977789 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:27.977844 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0806 06:26:27.977904 (+ 60us) server_negotiation.cc:299] Negotiation successful
0806 06:26:27.977946 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":119,"threads_started":1}
I20260806 06:26:27.982617 23737 tablet_bootstrap.cc:654] T 56f29a9251e748419c5a8e3941ae9104 P bdd4da71739a455c9495a8887b22fa09: Neither blocks nor log segments found. Creating new log.
I20260806 06:26:27.982504 23619 tablet_service.cc:1467] Processing CreateTablet for tablet 56f29a9251e748419c5a8e3941ae9104 (DEFAULT_TABLE table=test-table [id=7a300369a9244afbb12128bfdd868f22]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 06:26:27.982852 23619 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56f29a9251e748419c5a8e3941ae9104. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 06:26:27.982872 23737 log.cc:824] T 56f29a9251e748419c5a8e3941ae9104 P bdd4da71739a455c9495a8887b22fa09: Log is configured to *not* fsync() on all Append() calls
I20260806 06:26:27.982788 23215 tablet_service.cc:1467] Processing CreateTablet for tablet 56f29a9251e748419c5a8e3941ae9104 (DEFAULT_TABLE table=test-table [id=7a300369a9244afbb12128bfdd868f22]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 06:26:27.983167 23215 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56f29a9251e748419c5a8e3941ae9104. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 06:26:27.983595 23737 tablet_bootstrap.cc:492] T 56f29a9251e748419c5a8e3941ae9104 P bdd4da71739a455c9495a8887b22fa09: No bootstrap required, opened a new log
I20260806 06:26:27.983665 23737 ts_tablet_manager.cc:1397] T 56f29a9251e748419c5a8e3941ae9104 P bdd4da71739a455c9495a8887b22fa09: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260806 06:26:27.985127 23737 raft_consensus.cc:348] T 56f29a9251e748419c5a8e3941ae9104 P bdd4da71739a455c9495a8887b22fa09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4cd0bf3767a4024ba327faf3015304c" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 33499 } } peers { permanent_uuid: "279a4037d58b470d976dc6f43aff7790" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 39473 } } peers { permanent_uuid: "bdd4da71739a455c9495a8887b22fa09" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 33755 } }
I20260806 06:26:27.985294 23737 raft_consensus.cc:374] T 56f29a9251e748419c5a8e3941ae9104 P bdd4da71739a455c9495a8887b22fa09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 06:26:27.985342 23737 raft_consensus.cc:729] T 56f29a9251e748419c5a8e3941ae9104 P bdd4da71739a455c9495a8887b22fa09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bdd4da71739a455c9495a8887b22fa09, State: Initialized, Role: FOLLOWER
I20260806 06:26:27.985467 23737 consensus_queue.cc:260] T 56f29a9251e748419c5a8e3941ae9104 P bdd4da71739a455c9495a8887b22fa09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4cd0bf3767a4024ba327faf3015304c" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 33499 } } peers { permanent_uuid: "279a4037d58b470d976dc6f43aff7790" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 39473 } } peers { permanent_uuid: "bdd4da71739a455c9495a8887b22fa09" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 33755 } }
I20260806 06:26:27.985733 23737 ts_tablet_manager.cc:1428] T 56f29a9251e748419c5a8e3941ae9104 P bdd4da71739a455c9495a8887b22fa09: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260806 06:26:27.985886 23506 heartbeater.cc:503] Master 127.21.12.190:41067 was elected leader, sending a full tablet report...
I20260806 06:26:27.986006 23740 tablet_bootstrap.cc:492] T 56f29a9251e748419c5a8e3941ae9104 P 279a4037d58b470d976dc6f43aff7790: Bootstrap starting.
I20260806 06:26:27.986725 23742 tablet_bootstrap.cc:492] T 56f29a9251e748419c5a8e3941ae9104 P c4cd0bf3767a4024ba327faf3015304c: Bootstrap starting.
I20260806 06:26:27.987007 23740 tablet_bootstrap.cc:654] T 56f29a9251e748419c5a8e3941ae9104 P 279a4037d58b470d976dc6f43aff7790: Neither blocks nor log segments found. Creating new log.
I20260806 06:26:27.987340 23740 log.cc:824] T 56f29a9251e748419c5a8e3941ae9104 P 279a4037d58b470d976dc6f43aff7790: Log is configured to *not* fsync() on all Append() calls
I20260806 06:26:27.987366 23742 tablet_bootstrap.cc:654] T 56f29a9251e748419c5a8e3941ae9104 P c4cd0bf3767a4024ba327faf3015304c: Neither blocks nor log segments found. Creating new log.
I20260806 06:26:27.987700 23742 log.cc:824] T 56f29a9251e748419c5a8e3941ae9104 P c4cd0bf3767a4024ba327faf3015304c: Log is configured to *not* fsync() on all Append() calls
I20260806 06:26:27.988109 23740 tablet_bootstrap.cc:492] T 56f29a9251e748419c5a8e3941ae9104 P 279a4037d58b470d976dc6f43aff7790: No bootstrap required, opened a new log
I20260806 06:26:27.988186 23740 ts_tablet_manager.cc:1397] T 56f29a9251e748419c5a8e3941ae9104 P 279a4037d58b470d976dc6f43aff7790: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260806 06:26:27.988428 23742 tablet_bootstrap.cc:492] T 56f29a9251e748419c5a8e3941ae9104 P c4cd0bf3767a4024ba327faf3015304c: No bootstrap required, opened a new log
I20260806 06:26:27.988509 23742 ts_tablet_manager.cc:1397] T 56f29a9251e748419c5a8e3941ae9104 P c4cd0bf3767a4024ba327faf3015304c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260806 06:26:27.990214 23740 raft_consensus.cc:348] T 56f29a9251e748419c5a8e3941ae9104 P 279a4037d58b470d976dc6f43aff7790 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4cd0bf3767a4024ba327faf3015304c" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 33499 } } peers { permanent_uuid: "279a4037d58b470d976dc6f43aff7790" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 39473 } } peers { permanent_uuid: "bdd4da71739a455c9495a8887b22fa09" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 33755 } }
I20260806 06:26:27.990418 23740 raft_consensus.cc:374] T 56f29a9251e748419c5a8e3941ae9104 P 279a4037d58b470d976dc6f43aff7790 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 06:26:27.990470 23740 raft_consensus.cc:729] T 56f29a9251e748419c5a8e3941ae9104 P 279a4037d58b470d976dc6f43aff7790 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 279a4037d58b470d976dc6f43aff7790, State: Initialized, Role: FOLLOWER
I20260806 06:26:27.990594 23740 consensus_queue.cc:260] T 56f29a9251e748419c5a8e3941ae9104 P 279a4037d58b470d976dc6f43aff7790 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4cd0bf3767a4024ba327faf3015304c" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 33499 } } peers { permanent_uuid: "279a4037d58b470d976dc6f43aff7790" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 39473 } } peers { permanent_uuid: "bdd4da71739a455c9495a8887b22fa09" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 33755 } }
I20260806 06:26:27.990636 23742 raft_consensus.cc:348] T 56f29a9251e748419c5a8e3941ae9104 P c4cd0bf3767a4024ba327faf3015304c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4cd0bf3767a4024ba327faf3015304c" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 33499 } } peers { permanent_uuid: "279a4037d58b470d976dc6f43aff7790" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 39473 } } peers { permanent_uuid: "bdd4da71739a455c9495a8887b22fa09" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 33755 } }
I20260806 06:26:27.990788 23742 raft_consensus.cc:374] T 56f29a9251e748419c5a8e3941ae9104 P c4cd0bf3767a4024ba327faf3015304c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 06:26:27.990839 23742 raft_consensus.cc:729] T 56f29a9251e748419c5a8e3941ae9104 P c4cd0bf3767a4024ba327faf3015304c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4cd0bf3767a4024ba327faf3015304c, State: Initialized, Role: FOLLOWER
I20260806 06:26:27.990877 23740 ts_tablet_manager.cc:1428] T 56f29a9251e748419c5a8e3941ae9104 P 279a4037d58b470d976dc6f43aff7790: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260806 06:26:27.990909 23711 heartbeater.cc:503] Master 127.21.12.190:41067 was elected leader, sending a full tablet report...
I20260806 06:26:27.990957 23742 consensus_queue.cc:260] T 56f29a9251e748419c5a8e3941ae9104 P c4cd0bf3767a4024ba327faf3015304c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4cd0bf3767a4024ba327faf3015304c" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 33499 } } peers { permanent_uuid: "279a4037d58b470d976dc6f43aff7790" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 39473 } } peers { permanent_uuid: "bdd4da71739a455c9495a8887b22fa09" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 33755 } }
I20260806 06:26:27.991205 23742 ts_tablet_manager.cc:1428] T 56f29a9251e748419c5a8e3941ae9104 P c4cd0bf3767a4024ba327faf3015304c: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260806 06:26:27.991266 23305 heartbeater.cc:503] Master 127.21.12.190:41067 was elected leader, sending a full tablet report...
I20260806 06:26:27.996117 23739 raft_consensus.cc:482] T 56f29a9251e748419c5a8e3941ae9104 P bdd4da71739a455c9495a8887b22fa09 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 06:26:27.996203 23739 raft_consensus.cc:504] T 56f29a9251e748419c5a8e3941ae9104 P bdd4da71739a455c9495a8887b22fa09 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4cd0bf3767a4024ba327faf3015304c" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 33499 } } peers { permanent_uuid: "279a4037d58b470d976dc6f43aff7790" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 39473 } } peers { permanent_uuid: "bdd4da71739a455c9495a8887b22fa09" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 33755 } }
I20260806 06:26:27.996462 23739 leader_election.cc:288] T 56f29a9251e748419c5a8e3941ae9104 P bdd4da71739a455c9495a8887b22fa09 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c4cd0bf3767a4024ba327faf3015304c (127.21.12.129:33499), 279a4037d58b470d976dc6f43aff7790 (127.21.12.131:39473)
I20260806 06:26:27.999511 23346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:27.996658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.129:33499 (local address 127.21.12.130:34183)
0806 06:26:27.996709 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0806 06:26:27.996721 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 06:26:27.996763 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:26:27.996994 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:26:27.996995 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:26:27.996997 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 06:26:27.997064 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:27.997066 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:27.997771 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:27.997772 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:27.998609 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:27.998612 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:27.999274 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:27.999275 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:27.999367 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:27.999371 (+ 4us) client_negotiation.cc:769] Sending connection context
0806 06:26:27.999423 (+ 52us) client_negotiation.cc:240] Negotiation successful
0806 06:26:27.999461 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260806 06:26:27.999554 23735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:27.996712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.130:34183 (local address 127.21.12.129:33499)
0806 06:26:27.996754 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0806 06:26:27.996756 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:26:27.996765 (+ 9us) server_negotiation.cc:407] Connection header received
0806 06:26:27.996907 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:26:27.996907 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:26:27.996917 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:26:27.996977 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 06:26:27.997130 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:27.997710 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:27.998682 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:27.999168 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:27.999235 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:27.999243 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0806 06:26:27.999451 (+ 208us) server_negotiation.cc:299] Negotiation successful
0806 06:26:27.999495 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260806 06:26:27.999732 23747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:27.996774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.131:39473 (local address 127.21.12.130:56935)
0806 06:26:27.996934 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0806 06:26:27.996944 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0806 06:26:27.996997 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:26:27.997134 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:26:27.997134 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:26:27.997136 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 06:26:27.997203 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:27.997205 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:27.997931 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:27.997932 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:27.998711 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:27.998713 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:27.999415 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:27.999416 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:27.999492 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:27.999496 (+ 4us) client_negotiation.cc:769] Sending connection context
0806 06:26:27.999606 (+ 110us) client_negotiation.cc:240] Negotiation successful
0806 06:26:27.999647 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":113,"threads_started":1}
I20260806 06:26:27.999760 23734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:27.996832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.130:56935 (local address 127.21.12.131:39473)
0806 06:26:27.996878 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0806 06:26:27.996880 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:26:27.997000 (+ 120us) server_negotiation.cc:407] Connection header received
0806 06:26:27.997060 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:26:27.997060 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:26:27.997071 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:26:27.997130 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 06:26:27.997285 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:27.997857 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:27.998786 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:27.999318 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:27.999394 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:27.999402 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0806 06:26:27.999663 (+ 261us) server_negotiation.cc:299] Negotiation successful
0806 06:26:27.999701 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260806 06:26:27.999835 23240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56f29a9251e748419c5a8e3941ae9104" candidate_uuid: "bdd4da71739a455c9495a8887b22fa09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4cd0bf3767a4024ba327faf3015304c" is_pre_election: true
I20260806 06:26:27.999948 23240 raft_consensus.cc:2454] T 56f29a9251e748419c5a8e3941ae9104 P c4cd0bf3767a4024ba327faf3015304c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bdd4da71739a455c9495a8887b22fa09 in term 0.
I20260806 06:26:28.000164 23338 leader_election.cc:302] T 56f29a9251e748419c5a8e3941ae9104 P bdd4da71739a455c9495a8887b22fa09 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bdd4da71739a455c9495a8887b22fa09, c4cd0bf3767a4024ba327faf3015304c; no voters:
I20260806 06:26:28.000274 23643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56f29a9251e748419c5a8e3941ae9104" candidate_uuid: "bdd4da71739a455c9495a8887b22fa09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "279a4037d58b470d976dc6f43aff7790" is_pre_election: true
I20260806 06:26:28.000315 23739 raft_consensus.cc:2781] T 56f29a9251e748419c5a8e3941ae9104 P bdd4da71739a455c9495a8887b22fa09 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 06:26:28.000388 23739 raft_consensus.cc:482] T 56f29a9251e748419c5a8e3941ae9104 P bdd4da71739a455c9495a8887b22fa09 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 06:26:28.000397 23643 raft_consensus.cc:2454] T 56f29a9251e748419c5a8e3941ae9104 P 279a4037d58b470d976dc6f43aff7790 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bdd4da71739a455c9495a8887b22fa09 in term 0.
I20260806 06:26:28.000417 23739 raft_consensus.cc:3037] T 56f29a9251e748419c5a8e3941ae9104 P bdd4da71739a455c9495a8887b22fa09 [term 0 FOLLOWER]: Advancing to term 1
I20260806 06:26:28.001312 23739 raft_consensus.cc:504] T 56f29a9251e748419c5a8e3941ae9104 P bdd4da71739a455c9495a8887b22fa09 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4cd0bf3767a4024ba327faf3015304c" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 33499 } } peers { permanent_uuid: "279a4037d58b470d976dc6f43aff7790" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 39473 } } peers { permanent_uuid: "bdd4da71739a455c9495a8887b22fa09" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 33755 } }
I20260806 06:26:28.001425 23739 leader_election.cc:288] T 56f29a9251e748419c5a8e3941ae9104 P bdd4da71739a455c9495a8887b22fa09 [CANDIDATE]: Term 1 election: Requested vote from peers c4cd0bf3767a4024ba327faf3015304c (127.21.12.129:33499), 279a4037d58b470d976dc6f43aff7790 (127.21.12.131:39473)
I20260806 06:26:28.001618 23240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56f29a9251e748419c5a8e3941ae9104" candidate_uuid: "bdd4da71739a455c9495a8887b22fa09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4cd0bf3767a4024ba327faf3015304c"
I20260806 06:26:28.001639 23643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56f29a9251e748419c5a8e3941ae9104" candidate_uuid: "bdd4da71739a455c9495a8887b22fa09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "279a4037d58b470d976dc6f43aff7790"
I20260806 06:26:28.001693 23240 raft_consensus.cc:3037] T 56f29a9251e748419c5a8e3941ae9104 P c4cd0bf3767a4024ba327faf3015304c [term 0 FOLLOWER]: Advancing to term 1
I20260806 06:26:28.001703 23643 raft_consensus.cc:3037] T 56f29a9251e748419c5a8e3941ae9104 P 279a4037d58b470d976dc6f43aff7790 [term 0 FOLLOWER]: Advancing to term 1
I20260806 06:26:28.002612 23240 raft_consensus.cc:2454] T 56f29a9251e748419c5a8e3941ae9104 P c4cd0bf3767a4024ba327faf3015304c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bdd4da71739a455c9495a8887b22fa09 in term 1.
I20260806 06:26:28.002616 23643 raft_consensus.cc:2454] T 56f29a9251e748419c5a8e3941ae9104 P 279a4037d58b470d976dc6f43aff7790 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bdd4da71739a455c9495a8887b22fa09 in term 1.
I20260806 06:26:28.002763 23338 leader_election.cc:302] T 56f29a9251e748419c5a8e3941ae9104 P bdd4da71739a455c9495a8887b22fa09 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bdd4da71739a455c9495a8887b22fa09, c4cd0bf3767a4024ba327faf3015304c; no voters:
I20260806 06:26:28.002885 23739 raft_consensus.cc:2781] T 56f29a9251e748419c5a8e3941ae9104 P bdd4da71739a455c9495a8887b22fa09 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 06:26:28.003051 23739 raft_consensus.cc:686] T 56f29a9251e748419c5a8e3941ae9104 P bdd4da71739a455c9495a8887b22fa09 [term 1 LEADER]: Becoming Leader. State: Replica: bdd4da71739a455c9495a8887b22fa09, State: Running, Role: LEADER
I20260806 06:26:28.003140 23739 consensus_queue.cc:237] T 56f29a9251e748419c5a8e3941ae9104 P bdd4da71739a455c9495a8887b22fa09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4cd0bf3767a4024ba327faf3015304c" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 33499 } } peers { permanent_uuid: "279a4037d58b470d976dc6f43aff7790" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 39473 } } peers { permanent_uuid: "bdd4da71739a455c9495a8887b22fa09" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 33755 } }
I20260806 06:26:28.003930 23029 catalog_manager.cc:5495] T 56f29a9251e748419c5a8e3941ae9104 P bdd4da71739a455c9495a8887b22fa09 reported cstate change: term changed from 0 to 1, leader changed from <none> to bdd4da71739a455c9495a8887b22fa09 (127.21.12.130). New cstate: current_term: 1 leader_uuid: "bdd4da71739a455c9495a8887b22fa09" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4cd0bf3767a4024ba327faf3015304c" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 33499 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "279a4037d58b470d976dc6f43aff7790" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 39473 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bdd4da71739a455c9495a8887b22fa09" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 33755 } health_report { overall_health: HEALTHY } } }
I20260806 06:26:28.017462 23733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:28.014875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45994 (local address 127.21.12.130:33755)
0806 06:26:28.014921 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0806 06:26:28.014923 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:26:28.014934 (+ 11us) server_negotiation.cc:407] Connection header received
0806 06:26:28.015076 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:26:28.015077 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:26:28.015092 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:26:28.015211 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 06:26:28.015347 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:28.015800 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:28.016571 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:28.016753 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:28.016807 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:28.017005 (+ 198us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 06:26:28.017096 (+ 91us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 06:26:28.017163 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0806 06:26:28.017207 (+ 44us) server_negotiation.cc:299] Negotiation successful
0806 06:26:28.017227 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260806 06:26:28.019345 23643 raft_consensus.cc:1261] T 56f29a9251e748419c5a8e3941ae9104 P 279a4037d58b470d976dc6f43aff7790 [term 1 FOLLOWER]: Refusing update from remote peer bdd4da71739a455c9495a8887b22fa09: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 06:26:28.019435 23240 raft_consensus.cc:1261] T 56f29a9251e748419c5a8e3941ae9104 P c4cd0bf3767a4024ba327faf3015304c [term 1 FOLLOWER]: Refusing update from remote peer bdd4da71739a455c9495a8887b22fa09: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 06:26:28.019660 23739 consensus_queue.cc:1035] T 56f29a9251e748419c5a8e3941ae9104 P bdd4da71739a455c9495a8887b22fa09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "279a4037d58b470d976dc6f43aff7790" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 39473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 06:26:28.019907 23748 consensus_queue.cc:1035] T 56f29a9251e748419c5a8e3941ae9104 P bdd4da71739a455c9495a8887b22fa09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4cd0bf3767a4024ba327faf3015304c" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 33499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 06:26:28.022125 23751 mvcc.cc:205] Tried to move back new op lower bound from 7315446120524816384 to 7315446120461488128. Current Snapshot: MvccSnapshot[applied={T|T < 7315446120524816384}]
I20260806 06:26:28.022557 23755 mvcc.cc:205] Tried to move back new op lower bound from 7315446120524816384 to 7315446120461488128. Current Snapshot: MvccSnapshot[applied={T|T < 7315446120524816384}]
I20260806 06:26:28.025754 23029 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33586:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 06:26:28.025868 23029 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33586:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 06:26:28.026690 23029 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 31c5e8e609c44afa8e0f2b07d741d0af: Sending DeleteTablet for 3 replicas of tablet 56f29a9251e748419c5a8e3941ae9104
I20260806 06:26:28.027070 23411 tablet_service.cc:1514] Processing DeleteTablet for tablet 56f29a9251e748419c5a8e3941ae9104 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 06:26:28 UTC) from {username='kudu', principal='kudu/127.21.12.190@KRBTEST.COM'} at 127.0.0.1:45980
I20260806 06:26:28.027357 23029 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33586:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 06:26:28.027457 23029 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 06:26:28.028602 23619 tablet_service.cc:1514] Processing DeleteTablet for tablet 56f29a9251e748419c5a8e3941ae9104 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 06:26:28 UTC) from {username='kudu', principal='kudu/127.21.12.190@KRBTEST.COM'} at 127.0.0.1:56918
I20260806 06:26:28.028620 23214 tablet_service.cc:1467] Processing CreateTablet for tablet c07e37b5f6ac41e2b784acffa978ed25 (DEFAULT_TABLE table=test-table [id=9e140931887845f5b18bee384d60827d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 06:26:28.028764 23214 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c07e37b5f6ac41e2b784acffa978ed25. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 06:26:28.028808 23765 tablet_replica.cc:330] T 56f29a9251e748419c5a8e3941ae9104 P 279a4037d58b470d976dc6f43aff7790: stopping tablet replica
I20260806 06:26:28.028807 23215 tablet_service.cc:1514] Processing DeleteTablet for tablet 56f29a9251e748419c5a8e3941ae9104 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 06:26:28 UTC) from {username='kudu', principal='kudu/127.21.12.190@KRBTEST.COM'} at 127.0.0.1:40312
I20260806 06:26:28.028900 23765 raft_consensus.cc:2229] T 56f29a9251e748419c5a8e3941ae9104 P 279a4037d58b470d976dc6f43aff7790 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 06:26:28.029004 23765 raft_consensus.cc:2258] T 56f29a9251e748419c5a8e3941ae9104 P 279a4037d58b470d976dc6f43aff7790 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 06:26:28.029659 23619 tablet_service.cc:1467] Processing CreateTablet for tablet c07e37b5f6ac41e2b784acffa978ed25 (DEFAULT_TABLE table=test-table [id=9e140931887845f5b18bee384d60827d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 06:26:28.029742 23765 ts_tablet_manager.cc:1905] T 56f29a9251e748419c5a8e3941ae9104 P 279a4037d58b470d976dc6f43aff7790: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 06:26:28.029778 23619 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c07e37b5f6ac41e2b784acffa978ed25. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 06:26:28.031160 23740 tablet_bootstrap.cc:492] T c07e37b5f6ac41e2b784acffa978ed25 P 279a4037d58b470d976dc6f43aff7790: Bootstrap starting.
I20260806 06:26:28.031257 23742 tablet_bootstrap.cc:492] T c07e37b5f6ac41e2b784acffa978ed25 P c4cd0bf3767a4024ba327faf3015304c: Bootstrap starting.
I20260806 06:26:28.032100 23765 ts_tablet_manager.cc:1918] T 56f29a9251e748419c5a8e3941ae9104 P 279a4037d58b470d976dc6f43aff7790: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 06:26:28.032159 23765 log.cc:1196] T 56f29a9251e748419c5a8e3941ae9104 P 279a4037d58b470d976dc6f43aff7790: Deleting WAL directory at /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785997581049705-21554-0/minicluster-data/ts-2/wal/wals/56f29a9251e748419c5a8e3941ae9104
I20260806 06:26:28.032173 23740 tablet_bootstrap.cc:654] T c07e37b5f6ac41e2b784acffa978ed25 P 279a4037d58b470d976dc6f43aff7790: Neither blocks nor log segments found. Creating new log.
I20260806 06:26:28.032270 23742 tablet_bootstrap.cc:654] T c07e37b5f6ac41e2b784acffa978ed25 P c4cd0bf3767a4024ba327faf3015304c: Neither blocks nor log segments found. Creating new log.
I20260806 06:26:28.032402 23765 ts_tablet_manager.cc:1939] T 56f29a9251e748419c5a8e3941ae9104 P 279a4037d58b470d976dc6f43aff7790: Deleting consensus metadata
I20260806 06:26:28.032799 23012 catalog_manager.cc:4841] TS 279a4037d58b470d976dc6f43aff7790 (127.21.12.131:39473): tablet 56f29a9251e748419c5a8e3941ae9104 (table test-table [id=7a300369a9244afbb12128bfdd868f22]) successfully deleted
I20260806 06:26:28.033244 23740 tablet_bootstrap.cc:492] T c07e37b5f6ac41e2b784acffa978ed25 P 279a4037d58b470d976dc6f43aff7790: No bootstrap required, opened a new log
I20260806 06:26:28.033294 23740 ts_tablet_manager.cc:1397] T c07e37b5f6ac41e2b784acffa978ed25 P 279a4037d58b470d976dc6f43aff7790: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20260806 06:26:28.033330 23411 tablet_service.cc:1467] Processing CreateTablet for tablet c07e37b5f6ac41e2b784acffa978ed25 (DEFAULT_TABLE table=test-table [id=9e140931887845f5b18bee384d60827d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 06:26:28.033443 23766 tablet_replica.cc:330] T 56f29a9251e748419c5a8e3941ae9104 P c4cd0bf3767a4024ba327faf3015304c: stopping tablet replica
I20260806 06:26:28.033427 23740 raft_consensus.cc:348] T c07e37b5f6ac41e2b784acffa978ed25 P 279a4037d58b470d976dc6f43aff7790 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4cd0bf3767a4024ba327faf3015304c" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 33499 } } peers { permanent_uuid: "bdd4da71739a455c9495a8887b22fa09" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 33755 } } peers { permanent_uuid: "279a4037d58b470d976dc6f43aff7790" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 39473 } }
I20260806 06:26:28.033456 23411 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c07e37b5f6ac41e2b784acffa978ed25. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 06:26:28.033486 23740 raft_consensus.cc:374] T c07e37b5f6ac41e2b784acffa978ed25 P 279a4037d58b470d976dc6f43aff7790 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 06:26:28.033510 23740 raft_consensus.cc:729] T c07e37b5f6ac41e2b784acffa978ed25 P 279a4037d58b470d976dc6f43aff7790 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 279a4037d58b470d976dc6f43aff7790, State: Initialized, Role: FOLLOWER
I20260806 06:26:28.033550 23740 consensus_queue.cc:260] T c07e37b5f6ac41e2b784acffa978ed25 P 279a4037d58b470d976dc6f43aff7790 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4cd0bf3767a4024ba327faf3015304c" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 33499 } } peers { permanent_uuid: "bdd4da71739a455c9495a8887b22fa09" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 33755 } } peers { permanent_uuid: "279a4037d58b470d976dc6f43aff7790" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 39473 } }
I20260806 06:26:28.033619 23740 ts_tablet_manager.cc:1428] T c07e37b5f6ac41e2b784acffa978ed25 P 279a4037d58b470d976dc6f43aff7790: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260806 06:26:28.033756 23766 raft_consensus.cc:2229] T 56f29a9251e748419c5a8e3941ae9104 P c4cd0bf3767a4024ba327faf3015304c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 06:26:28.033903 23766 raft_consensus.cc:2258] T 56f29a9251e748419c5a8e3941ae9104 P c4cd0bf3767a4024ba327faf3015304c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 06:26:28.034222 23764 tablet_replica.cc:330] T 56f29a9251e748419c5a8e3941ae9104 P bdd4da71739a455c9495a8887b22fa09: stopping tablet replica
I20260806 06:26:28.034407 23764 raft_consensus.cc:2229] T 56f29a9251e748419c5a8e3941ae9104 P bdd4da71739a455c9495a8887b22fa09 [term 1 LEADER]: Raft consensus shutting down.
I20260806 06:26:28.034598 23764 raft_consensus.cc:2258] T 56f29a9251e748419c5a8e3941ae9104 P bdd4da71739a455c9495a8887b22fa09 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 06:26:28.034816 23766 ts_tablet_manager.cc:1905] T 56f29a9251e748419c5a8e3941ae9104 P c4cd0bf3767a4024ba327faf3015304c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 06:26:28.035084 23764 ts_tablet_manager.cc:1905] T 56f29a9251e748419c5a8e3941ae9104 P bdd4da71739a455c9495a8887b22fa09: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 06:26:28.035324 23737 tablet_bootstrap.cc:492] T c07e37b5f6ac41e2b784acffa978ed25 P bdd4da71739a455c9495a8887b22fa09: Bootstrap starting.
I20260806 06:26:28.036338 23737 tablet_bootstrap.cc:654] T c07e37b5f6ac41e2b784acffa978ed25 P bdd4da71739a455c9495a8887b22fa09: Neither blocks nor log segments found. Creating new log.
I20260806 06:26:28.037002 23766 ts_tablet_manager.cc:1918] T 56f29a9251e748419c5a8e3941ae9104 P c4cd0bf3767a4024ba327faf3015304c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 06:26:28.037091 23766 log.cc:1196] T 56f29a9251e748419c5a8e3941ae9104 P c4cd0bf3767a4024ba327faf3015304c: Deleting WAL directory at /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785997581049705-21554-0/minicluster-data/ts-0/wal/wals/56f29a9251e748419c5a8e3941ae9104
I20260806 06:26:28.037406 23766 ts_tablet_manager.cc:1939] T 56f29a9251e748419c5a8e3941ae9104 P c4cd0bf3767a4024ba327faf3015304c: Deleting consensus metadata
I20260806 06:26:28.037623 23764 ts_tablet_manager.cc:1918] T 56f29a9251e748419c5a8e3941ae9104 P bdd4da71739a455c9495a8887b22fa09: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 06:26:28.037667 23737 tablet_bootstrap.cc:492] T c07e37b5f6ac41e2b784acffa978ed25 P bdd4da71739a455c9495a8887b22fa09: No bootstrap required, opened a new log
I20260806 06:26:28.037755 23737 ts_tablet_manager.cc:1397] T c07e37b5f6ac41e2b784acffa978ed25 P bdd4da71739a455c9495a8887b22fa09: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260806 06:26:28.037779 23012 catalog_manager.cc:4841] TS c4cd0bf3767a4024ba327faf3015304c (127.21.12.129:33499): tablet 56f29a9251e748419c5a8e3941ae9104 (table test-table [id=7a300369a9244afbb12128bfdd868f22]) successfully deleted
I20260806 06:26:28.037823 23764 log.cc:1196] T 56f29a9251e748419c5a8e3941ae9104 P bdd4da71739a455c9495a8887b22fa09: Deleting WAL directory at /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785997581049705-21554-0/minicluster-data/ts-1/wal/wals/56f29a9251e748419c5a8e3941ae9104
I20260806 06:26:28.038089 23764 ts_tablet_manager.cc:1939] T 56f29a9251e748419c5a8e3941ae9104 P bdd4da71739a455c9495a8887b22fa09: Deleting consensus metadata
I20260806 06:26:28.038093 23737 raft_consensus.cc:348] T c07e37b5f6ac41e2b784acffa978ed25 P bdd4da71739a455c9495a8887b22fa09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4cd0bf3767a4024ba327faf3015304c" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 33499 } } peers { permanent_uuid: "bdd4da71739a455c9495a8887b22fa09" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 33755 } } peers { permanent_uuid: "279a4037d58b470d976dc6f43aff7790" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 39473 } }
I20260806 06:26:28.038223 23737 raft_consensus.cc:374] T c07e37b5f6ac41e2b784acffa978ed25 P bdd4da71739a455c9495a8887b22fa09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 06:26:28.038242 23737 raft_consensus.cc:729] T c07e37b5f6ac41e2b784acffa978ed25 P bdd4da71739a455c9495a8887b22fa09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bdd4da71739a455c9495a8887b22fa09, State: Initialized, Role: FOLLOWER
I20260806 06:26:28.038311 23737 consensus_queue.cc:260] T c07e37b5f6ac41e2b784acffa978ed25 P bdd4da71739a455c9495a8887b22fa09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4cd0bf3767a4024ba327faf3015304c" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 33499 } } peers { permanent_uuid: "bdd4da71739a455c9495a8887b22fa09" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 33755 } } peers { permanent_uuid: "279a4037d58b470d976dc6f43aff7790" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 39473 } }
I20260806 06:26:28.038390 23737 ts_tablet_manager.cc:1428] T c07e37b5f6ac41e2b784acffa978ed25 P bdd4da71739a455c9495a8887b22fa09: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 06:26:28.038493 23014 catalog_manager.cc:4841] TS bdd4da71739a455c9495a8887b22fa09 (127.21.12.130:33755): tablet 56f29a9251e748419c5a8e3941ae9104 (table test-table [id=7a300369a9244afbb12128bfdd868f22]) successfully deleted
I20260806 06:26:28.038503 23742 tablet_bootstrap.cc:492] T c07e37b5f6ac41e2b784acffa978ed25 P c4cd0bf3767a4024ba327faf3015304c: No bootstrap required, opened a new log
I20260806 06:26:28.038551 23742 ts_tablet_manager.cc:1397] T c07e37b5f6ac41e2b784acffa978ed25 P c4cd0bf3767a4024ba327faf3015304c: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20260806 06:26:28.038975 23742 raft_consensus.cc:348] T c07e37b5f6ac41e2b784acffa978ed25 P c4cd0bf3767a4024ba327faf3015304c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4cd0bf3767a4024ba327faf3015304c" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 33499 } } peers { permanent_uuid: "bdd4da71739a455c9495a8887b22fa09" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 33755 } } peers { permanent_uuid: "279a4037d58b470d976dc6f43aff7790" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 39473 } }
I20260806 06:26:28.039062 23742 raft_consensus.cc:374] T c07e37b5f6ac41e2b784acffa978ed25 P c4cd0bf3767a4024ba327faf3015304c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 06:26:28.039084 23742 raft_consensus.cc:729] T c07e37b5f6ac41e2b784acffa978ed25 P c4cd0bf3767a4024ba327faf3015304c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4cd0bf3767a4024ba327faf3015304c, State: Initialized, Role: FOLLOWER
I20260806 06:26:28.039131 23742 consensus_queue.cc:260] T c07e37b5f6ac41e2b784acffa978ed25 P c4cd0bf3767a4024ba327faf3015304c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4cd0bf3767a4024ba327faf3015304c" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 33499 } } peers { permanent_uuid: "bdd4da71739a455c9495a8887b22fa09" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 33755 } } peers { permanent_uuid: "279a4037d58b470d976dc6f43aff7790" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 39473 } }
I20260806 06:26:28.039222 23742 ts_tablet_manager.cc:1428] T c07e37b5f6ac41e2b784acffa978ed25 P c4cd0bf3767a4024ba327faf3015304c: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260806 06:26:28.051993 23745 raft_consensus.cc:482] T c07e37b5f6ac41e2b784acffa978ed25 P 279a4037d58b470d976dc6f43aff7790 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 06:26:28.052093 23745 raft_consensus.cc:504] T c07e37b5f6ac41e2b784acffa978ed25 P 279a4037d58b470d976dc6f43aff7790 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4cd0bf3767a4024ba327faf3015304c" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 33499 } } peers { permanent_uuid: "bdd4da71739a455c9495a8887b22fa09" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 33755 } } peers { permanent_uuid: "279a4037d58b470d976dc6f43aff7790" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 39473 } }
I20260806 06:26:28.052418 23745 leader_election.cc:288] T c07e37b5f6ac41e2b784acffa978ed25 P 279a4037d58b470d976dc6f43aff7790 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c4cd0bf3767a4024ba327faf3015304c (127.21.12.129:33499), bdd4da71739a455c9495a8887b22fa09 (127.21.12.130:33755)
I20260806 06:26:28.056015 23552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:28.052563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.129:33499 (local address 127.21.12.131:34463)
0806 06:26:28.052620 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0806 06:26:28.052632 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 06:26:28.052674 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:26:28.052875 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:26:28.052876 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:26:28.052878 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 06:26:28.052950 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:28.052952 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:28.053673 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:28.053674 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:28.054819 (+ 1145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:28.054824 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:28.055598 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:28.055599 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:28.055751 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:28.055757 (+ 6us) client_negotiation.cc:769] Sending connection context
0806 06:26:28.055815 (+ 58us) client_negotiation.cc:240] Negotiation successful
0806 06:26:28.055859 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260806 06:26:28.056134 23735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:28.052633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.131:34463 (local address 127.21.12.129:33499)
0806 06:26:28.052675 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0806 06:26:28.052677 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:26:28.052686 (+ 9us) server_negotiation.cc:407] Connection header received
0806 06:26:28.052762 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:26:28.052763 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:26:28.052774 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:26:28.052827 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 06:26:28.053049 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:28.053588 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:28.054935 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:28.055511 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:28.055582 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:28.055593 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0806 06:26:28.056018 (+ 425us) server_negotiation.cc:299] Negotiation successful
0806 06:26:28.056065 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260806 06:26:28.056466 23240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c07e37b5f6ac41e2b784acffa978ed25" candidate_uuid: "279a4037d58b470d976dc6f43aff7790" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4cd0bf3767a4024ba327faf3015304c" is_pre_election: true
I20260806 06:26:28.056582 23240 raft_consensus.cc:2454] T c07e37b5f6ac41e2b784acffa978ed25 P c4cd0bf3767a4024ba327faf3015304c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 279a4037d58b470d976dc6f43aff7790 in term 0.
I20260806 06:26:28.056798 23733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:28.052804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.131:48661 (local address 127.21.12.130:33755)
0806 06:26:28.052846 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0806 06:26:28.052848 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:26:28.052881 (+ 33us) server_negotiation.cc:407] Connection header received
0806 06:26:28.052905 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:26:28.052906 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:26:28.052915 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:26:28.052965 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 06:26:28.053171 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:28.053927 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:28.055512 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:28.056005 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:28.056080 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:28.056089 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0806 06:26:28.056687 (+ 598us) server_negotiation.cc:299] Negotiation successful
0806 06:26:28.056728 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260806 06:26:28.056818 23544 leader_election.cc:302] T c07e37b5f6ac41e2b784acffa978ed25 P 279a4037d58b470d976dc6f43aff7790 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 279a4037d58b470d976dc6f43aff7790, c4cd0bf3767a4024ba327faf3015304c; no voters:
I20260806 06:26:28.056947 23745 raft_consensus.cc:2781] T c07e37b5f6ac41e2b784acffa978ed25 P 279a4037d58b470d976dc6f43aff7790 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 06:26:28.056998 23745 raft_consensus.cc:482] T c07e37b5f6ac41e2b784acffa978ed25 P 279a4037d58b470d976dc6f43aff7790 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 06:26:28.057030 23745 raft_consensus.cc:3037] T c07e37b5f6ac41e2b784acffa978ed25 P 279a4037d58b470d976dc6f43aff7790 [term 0 FOLLOWER]: Advancing to term 1
I20260806 06:26:28.057148 23774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:28.052576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.130:33755 (local address 127.21.12.131:48661)
0806 06:26:28.052787 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0806 06:26:28.052797 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0806 06:26:28.052843 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:26:28.052968 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:26:28.052969 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:26:28.052971 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 06:26:28.053102 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:28.053104 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:28.054040 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:28.054041 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:28.055411 (+ 1370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:28.055415 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:28.056420 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:28.056421 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:28.056576 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:28.056582 (+ 6us) client_negotiation.cc:769] Sending connection context
0806 06:26:28.056916 (+ 334us) client_negotiation.cc:240] Negotiation successful
0806 06:26:28.056965 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":76,"threads_started":1}
I20260806 06:26:28.057515 23435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c07e37b5f6ac41e2b784acffa978ed25" candidate_uuid: "279a4037d58b470d976dc6f43aff7790" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bdd4da71739a455c9495a8887b22fa09" is_pre_election: true
I20260806 06:26:28.057653 23435 raft_consensus.cc:2454] T c07e37b5f6ac41e2b784acffa978ed25 P bdd4da71739a455c9495a8887b22fa09 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 279a4037d58b470d976dc6f43aff7790 in term 0.
I20260806 06:26:28.058255 23745 raft_consensus.cc:504] T c07e37b5f6ac41e2b784acffa978ed25 P 279a4037d58b470d976dc6f43aff7790 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4cd0bf3767a4024ba327faf3015304c" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 33499 } } peers { permanent_uuid: "bdd4da71739a455c9495a8887b22fa09" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 33755 } } peers { permanent_uuid: "279a4037d58b470d976dc6f43aff7790" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 39473 } }
I20260806 06:26:28.058431 23745 leader_election.cc:288] T c07e37b5f6ac41e2b784acffa978ed25 P 279a4037d58b470d976dc6f43aff7790 [CANDIDATE]: Term 1 election: Requested vote from peers c4cd0bf3767a4024ba327faf3015304c (127.21.12.129:33499), bdd4da71739a455c9495a8887b22fa09 (127.21.12.130:33755)
I20260806 06:26:28.058600 23240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c07e37b5f6ac41e2b784acffa978ed25" candidate_uuid: "279a4037d58b470d976dc6f43aff7790" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4cd0bf3767a4024ba327faf3015304c"
I20260806 06:26:28.058682 23240 raft_consensus.cc:3037] T c07e37b5f6ac41e2b784acffa978ed25 P c4cd0bf3767a4024ba327faf3015304c [term 0 FOLLOWER]: Advancing to term 1
I20260806 06:26:28.058712 23435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c07e37b5f6ac41e2b784acffa978ed25" candidate_uuid: "279a4037d58b470d976dc6f43aff7790" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bdd4da71739a455c9495a8887b22fa09"
I20260806 06:26:28.058791 23435 raft_consensus.cc:3037] T c07e37b5f6ac41e2b784acffa978ed25 P bdd4da71739a455c9495a8887b22fa09 [term 0 FOLLOWER]: Advancing to term 1
I20260806 06:26:28.059724 23240 raft_consensus.cc:2454] T c07e37b5f6ac41e2b784acffa978ed25 P c4cd0bf3767a4024ba327faf3015304c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 279a4037d58b470d976dc6f43aff7790 in term 1.
I20260806 06:26:28.059880 23544 leader_election.cc:302] T c07e37b5f6ac41e2b784acffa978ed25 P 279a4037d58b470d976dc6f43aff7790 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 279a4037d58b470d976dc6f43aff7790, c4cd0bf3767a4024ba327faf3015304c; no voters:
I20260806 06:26:28.059965 23745 raft_consensus.cc:2781] T c07e37b5f6ac41e2b784acffa978ed25 P 279a4037d58b470d976dc6f43aff7790 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 06:26:28.059999 23435 raft_consensus.cc:2454] T c07e37b5f6ac41e2b784acffa978ed25 P bdd4da71739a455c9495a8887b22fa09 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 279a4037d58b470d976dc6f43aff7790 in term 1.
I20260806 06:26:28.060115 23745 raft_consensus.cc:686] T c07e37b5f6ac41e2b784acffa978ed25 P 279a4037d58b470d976dc6f43aff7790 [term 1 LEADER]: Becoming Leader. State: Replica: 279a4037d58b470d976dc6f43aff7790, State: Running, Role: LEADER
I20260806 06:26:28.060184 23745 consensus_queue.cc:237] T c07e37b5f6ac41e2b784acffa978ed25 P 279a4037d58b470d976dc6f43aff7790 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4cd0bf3767a4024ba327faf3015304c" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 33499 } } peers { permanent_uuid: "bdd4da71739a455c9495a8887b22fa09" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 33755 } } peers { permanent_uuid: "279a4037d58b470d976dc6f43aff7790" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 39473 } }
I20260806 06:26:28.060870 23031 catalog_manager.cc:5495] T c07e37b5f6ac41e2b784acffa978ed25 P 279a4037d58b470d976dc6f43aff7790 reported cstate change: term changed from 0 to 1, leader changed from <none> to 279a4037d58b470d976dc6f43aff7790 (127.21.12.131). New cstate: current_term: 1 leader_uuid: "279a4037d58b470d976dc6f43aff7790" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4cd0bf3767a4024ba327faf3015304c" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 33499 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bdd4da71739a455c9495a8887b22fa09" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 33755 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "279a4037d58b470d976dc6f43aff7790" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 39473 } health_report { overall_health: HEALTHY } } }
Aug 06 06:26:28 dist-test-slave-4tdj krb5kdc[22970](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997587, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.190@KRBTEST.COM for kudu/127.21.12.190@KRBTEST.COM
I20260806 06:26:28.103257 23096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:28.095232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33588 (local address 127.21.12.190:41067)
0806 06:26:28.095296 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0806 06:26:28.095298 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:26:28.095536 (+ 238us) server_negotiation.cc:407] Connection header received
0806 06:26:28.095595 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:26:28.095596 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:26:28.095612 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:26:28.095684 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:26:28.096339 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:28.096833 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:28.097851 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:28.098068 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:28.098138 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:28.100600 (+ 2462us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:26:28.100601 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:26:28.100602 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 06:26:28.100608 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:26:28.102038 (+ 1430us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:26:28.102514 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:26:28.102515 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:26:28.102516 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:26:28.102579 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:26:28.102789 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:26:28.102790 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:26:28.102790 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:26:28.102891 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:26:28.102940 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0806 06:26:28.103118 (+ 178us) server_negotiation.cc:299] Negotiation successful
0806 06:26:28.103190 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260806 06:26:28.103412 23784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:28.095161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:41067 (local address 127.0.0.1:33588)
0806 06:26:28.095434 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0806 06:26:28.095447 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 06:26:28.095533 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:26:28.095710 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:26:28.095711 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:26:28.096203 (+ 492us) client_negotiation.cc:189] Negotiated authn=SASL
0806 06:26:28.096263 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:28.096266 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:28.096917 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:28.096918 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:28.097757 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:28.097759 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:28.098149 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:28.098150 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:28.098209 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:28.098798 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 06:26:28.098803 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0806 06:26:28.100517 (+ 1714us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 06:26:28.102127 (+ 1610us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:26:28.102135 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:26:28.102137 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:26:28.102430 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:26:28.102644 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:26:28.102645 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:26:28.102645 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:26:28.102722 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:26:28.102948 (+ 226us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 06:26:28.102956 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 06:26:28.103019 (+ 63us) client_negotiation.cc:769] Sending connection context
0806 06:26:28.103101 (+ 82us) client_negotiation.cc:240] Negotiation successful
0806 06:26:28.103260 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":215,"mutex_wait_us":48,"thread_start_us":165,"threads_started":1}
I20260806 06:26:28.105808 23031 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.21.12.190@KRBTEST.COM'} at 127.0.0.1:33588:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260806 06:26:28.106118 23031 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 06:26:28.107496 23619 tablet_service.cc:1467] Processing CreateTablet for tablet a4b5f07ac5054118a33ed7247abc4a51 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=f136ab719ede4f42b0ca2838cf9d14ad]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 06:26:28.107654 23619 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4b5f07ac5054118a33ed7247abc4a51. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 06:26:28.107609 23214 tablet_service.cc:1467] Processing CreateTablet for tablet a4b5f07ac5054118a33ed7247abc4a51 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=f136ab719ede4f42b0ca2838cf9d14ad]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 06:26:28.107767 23214 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4b5f07ac5054118a33ed7247abc4a51. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 06:26:28.109087 23740 tablet_bootstrap.cc:492] T a4b5f07ac5054118a33ed7247abc4a51 P 279a4037d58b470d976dc6f43aff7790: Bootstrap starting.
I20260806 06:26:28.109642 23742 tablet_bootstrap.cc:492] T a4b5f07ac5054118a33ed7247abc4a51 P c4cd0bf3767a4024ba327faf3015304c: Bootstrap starting.
I20260806 06:26:28.109710 23411 tablet_service.cc:1467] Processing CreateTablet for tablet a4b5f07ac5054118a33ed7247abc4a51 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=f136ab719ede4f42b0ca2838cf9d14ad]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 06:26:28.109858 23411 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4b5f07ac5054118a33ed7247abc4a51. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 06:26:28.110001 23740 tablet_bootstrap.cc:654] T a4b5f07ac5054118a33ed7247abc4a51 P 279a4037d58b470d976dc6f43aff7790: Neither blocks nor log segments found. Creating new log.
I20260806 06:26:28.110817 23742 tablet_bootstrap.cc:654] T a4b5f07ac5054118a33ed7247abc4a51 P c4cd0bf3767a4024ba327faf3015304c: Neither blocks nor log segments found. Creating new log.
I20260806 06:26:28.111652 23737 tablet_bootstrap.cc:492] T a4b5f07ac5054118a33ed7247abc4a51 P bdd4da71739a455c9495a8887b22fa09: Bootstrap starting.
I20260806 06:26:28.112061 23742 tablet_bootstrap.cc:492] T a4b5f07ac5054118a33ed7247abc4a51 P c4cd0bf3767a4024ba327faf3015304c: No bootstrap required, opened a new log
I20260806 06:26:28.112107 23742 ts_tablet_manager.cc:1397] T a4b5f07ac5054118a33ed7247abc4a51 P c4cd0bf3767a4024ba327faf3015304c: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260806 06:26:28.112355 23742 raft_consensus.cc:348] T a4b5f07ac5054118a33ed7247abc4a51 P c4cd0bf3767a4024ba327faf3015304c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4cd0bf3767a4024ba327faf3015304c" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 33499 } } peers { permanent_uuid: "bdd4da71739a455c9495a8887b22fa09" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 33755 } } peers { permanent_uuid: "279a4037d58b470d976dc6f43aff7790" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 39473 } }
I20260806 06:26:28.112422 23742 raft_consensus.cc:374] T a4b5f07ac5054118a33ed7247abc4a51 P c4cd0bf3767a4024ba327faf3015304c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 06:26:28.112442 23742 raft_consensus.cc:729] T a4b5f07ac5054118a33ed7247abc4a51 P c4cd0bf3767a4024ba327faf3015304c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4cd0bf3767a4024ba327faf3015304c, State: Initialized, Role: FOLLOWER
I20260806 06:26:28.112486 23742 consensus_queue.cc:260] T a4b5f07ac5054118a33ed7247abc4a51 P c4cd0bf3767a4024ba327faf3015304c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4cd0bf3767a4024ba327faf3015304c" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 33499 } } peers { permanent_uuid: "bdd4da71739a455c9495a8887b22fa09" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 33755 } } peers { permanent_uuid: "279a4037d58b470d976dc6f43aff7790" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 39473 } }
I20260806 06:26:28.112581 23742 ts_tablet_manager.cc:1428] T a4b5f07ac5054118a33ed7247abc4a51 P c4cd0bf3767a4024ba327faf3015304c: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260806 06:26:28.112660 23737 tablet_bootstrap.cc:654] T a4b5f07ac5054118a33ed7247abc4a51 P bdd4da71739a455c9495a8887b22fa09: Neither blocks nor log segments found. Creating new log.
I20260806 06:26:28.112758 23746 tablet_replica.cc:439] T a4b5f07ac5054118a33ed7247abc4a51 P c4cd0bf3767a4024ba327faf3015304c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4cd0bf3767a4024ba327faf3015304c" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 33499 } } peers { permanent_uuid: "bdd4da71739a455c9495a8887b22fa09" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 33755 } } peers { permanent_uuid: "279a4037d58b470d976dc6f43aff7790" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 39473 } } }
I20260806 06:26:28.112841 23746 tablet_replica.cc:442] T a4b5f07ac5054118a33ed7247abc4a51 P c4cd0bf3767a4024ba327faf3015304c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 06:26:28.113270 23740 tablet_bootstrap.cc:492] T a4b5f07ac5054118a33ed7247abc4a51 P 279a4037d58b470d976dc6f43aff7790: No bootstrap required, opened a new log
I20260806 06:26:28.113322 23740 ts_tablet_manager.cc:1397] T a4b5f07ac5054118a33ed7247abc4a51 P 279a4037d58b470d976dc6f43aff7790: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20260806 06:26:28.113754 23737 tablet_bootstrap.cc:492] T a4b5f07ac5054118a33ed7247abc4a51 P bdd4da71739a455c9495a8887b22fa09: No bootstrap required, opened a new log
I20260806 06:26:28.113722 23740 raft_consensus.cc:348] T a4b5f07ac5054118a33ed7247abc4a51 P 279a4037d58b470d976dc6f43aff7790 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4cd0bf3767a4024ba327faf3015304c" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 33499 } } peers { permanent_uuid: "bdd4da71739a455c9495a8887b22fa09" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 33755 } } peers { permanent_uuid: "279a4037d58b470d976dc6f43aff7790" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 39473 } }
I20260806 06:26:28.113801 23740 raft_consensus.cc:374] T a4b5f07ac5054118a33ed7247abc4a51 P 279a4037d58b470d976dc6f43aff7790 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 06:26:28.113799 23737 ts_tablet_manager.cc:1397] T a4b5f07ac5054118a33ed7247abc4a51 P bdd4da71739a455c9495a8887b22fa09: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260806 06:26:28.113816 23740 raft_consensus.cc:729] T a4b5f07ac5054118a33ed7247abc4a51 P 279a4037d58b470d976dc6f43aff7790 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 279a4037d58b470d976dc6f43aff7790, State: Initialized, Role: FOLLOWER
I20260806 06:26:28.113852 23740 consensus_queue.cc:260] T a4b5f07ac5054118a33ed7247abc4a51 P 279a4037d58b470d976dc6f43aff7790 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4cd0bf3767a4024ba327faf3015304c" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 33499 } } peers { permanent_uuid: "bdd4da71739a455c9495a8887b22fa09" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 33755 } } peers { permanent_uuid: "279a4037d58b470d976dc6f43aff7790" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 39473 } }
I20260806 06:26:28.113937 23740 ts_tablet_manager.cc:1428] T a4b5f07ac5054118a33ed7247abc4a51 P 279a4037d58b470d976dc6f43aff7790: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260806 06:26:28.113983 23745 tablet_replica.cc:439] T a4b5f07ac5054118a33ed7247abc4a51 P 279a4037d58b470d976dc6f43aff7790: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4cd0bf3767a4024ba327faf3015304c" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 33499 } } peers { permanent_uuid: "bdd4da71739a455c9495a8887b22fa09" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 33755 } } peers { permanent_uuid: "279a4037d58b470d976dc6f43aff7790" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 39473 } } }
I20260806 06:26:28.114051 23745 tablet_replica.cc:442] T a4b5f07ac5054118a33ed7247abc4a51 P 279a4037d58b470d976dc6f43aff7790: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 06:26:28.114259 23737 raft_consensus.cc:348] T a4b5f07ac5054118a33ed7247abc4a51 P bdd4da71739a455c9495a8887b22fa09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4cd0bf3767a4024ba327faf3015304c" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 33499 } } peers { permanent_uuid: "bdd4da71739a455c9495a8887b22fa09" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 33755 } } peers { permanent_uuid: "279a4037d58b470d976dc6f43aff7790" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 39473 } }
I20260806 06:26:28.114382 23737 raft_consensus.cc:374] T a4b5f07ac5054118a33ed7247abc4a51 P bdd4da71739a455c9495a8887b22fa09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 06:26:28.114441 23737 raft_consensus.cc:729] T a4b5f07ac5054118a33ed7247abc4a51 P bdd4da71739a455c9495a8887b22fa09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bdd4da71739a455c9495a8887b22fa09, State: Initialized, Role: FOLLOWER
I20260806 06:26:28.114492 23737 consensus_queue.cc:260] T a4b5f07ac5054118a33ed7247abc4a51 P bdd4da71739a455c9495a8887b22fa09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4cd0bf3767a4024ba327faf3015304c" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 33499 } } peers { permanent_uuid: "bdd4da71739a455c9495a8887b22fa09" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 33755 } } peers { permanent_uuid: "279a4037d58b470d976dc6f43aff7790" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 39473 } }
I20260806 06:26:28.114594 23737 ts_tablet_manager.cc:1428] T a4b5f07ac5054118a33ed7247abc4a51 P bdd4da71739a455c9495a8887b22fa09: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 06:26:28.114622 23748 tablet_replica.cc:439] T a4b5f07ac5054118a33ed7247abc4a51 P bdd4da71739a455c9495a8887b22fa09: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4cd0bf3767a4024ba327faf3015304c" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 33499 } } peers { permanent_uuid: "bdd4da71739a455c9495a8887b22fa09" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 33755 } } peers { permanent_uuid: "279a4037d58b470d976dc6f43aff7790" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 39473 } } }
I20260806 06:26:28.114720 23748 tablet_replica.cc:442] T a4b5f07ac5054118a33ed7247abc4a51 P bdd4da71739a455c9495a8887b22fa09: This TxnStatusTablet replica's current role is: FOLLOWER
W20260806 06:26:28.133930 23306 tablet.cc:2369] T a4b5f07ac5054118a33ed7247abc4a51 P c4cd0bf3767a4024ba327faf3015304c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 06:26:28.185441 23712 tablet.cc:2369] T a4b5f07ac5054118a33ed7247abc4a51 P 279a4037d58b470d976dc6f43aff7790: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 06:26:28.206331 23746 raft_consensus.cc:482] T a4b5f07ac5054118a33ed7247abc4a51 P c4cd0bf3767a4024ba327faf3015304c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 06:26:28.206432 23746 raft_consensus.cc:504] T a4b5f07ac5054118a33ed7247abc4a51 P c4cd0bf3767a4024ba327faf3015304c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4cd0bf3767a4024ba327faf3015304c" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 33499 } } peers { permanent_uuid: "bdd4da71739a455c9495a8887b22fa09" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 33755 } } peers { permanent_uuid: "279a4037d58b470d976dc6f43aff7790" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 39473 } }
I20260806 06:26:28.206866 23746 leader_election.cc:288] T a4b5f07ac5054118a33ed7247abc4a51 P c4cd0bf3767a4024ba327faf3015304c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bdd4da71739a455c9495a8887b22fa09 (127.21.12.130:33755), 279a4037d58b470d976dc6f43aff7790 (127.21.12.131:39473)
I20260806 06:26:28.210047 23734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:28.207117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.129:59873 (local address 127.21.12.131:39473)
0806 06:26:28.207204 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0806 06:26:28.207206 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:26:28.207233 (+ 27us) server_negotiation.cc:407] Connection header received
0806 06:26:28.207319 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:26:28.207320 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:26:28.207330 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:26:28.207377 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 06:26:28.207528 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:28.208132 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:28.209164 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:28.209722 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:28.209778 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:28.209787 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0806 06:26:28.209940 (+ 153us) server_negotiation.cc:299] Negotiation successful
0806 06:26:28.209978 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260806 06:26:28.210076 23791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:28.207029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.131:39473 (local address 127.21.12.129:59873)
0806 06:26:28.207166 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0806 06:26:28.207177 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0806 06:26:28.207248 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:26:28.207386 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:26:28.207387 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:26:28.207390 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 06:26:28.207463 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:28.207465 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:28.208200 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:28.208201 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:28.209073 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:28.209075 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:28.209795 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:28.209796 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:28.209868 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:28.209873 (+ 5us) client_negotiation.cc:769] Sending connection context
0806 06:26:28.209922 (+ 49us) client_negotiation.cc:240] Negotiation successful
0806 06:26:28.209964 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":70,"threads_started":1}
I20260806 06:26:28.210320 23643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4b5f07ac5054118a33ed7247abc4a51" candidate_uuid: "c4cd0bf3767a4024ba327faf3015304c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "279a4037d58b470d976dc6f43aff7790" is_pre_election: true
I20260806 06:26:28.210417 23643 raft_consensus.cc:2454] T a4b5f07ac5054118a33ed7247abc4a51 P 279a4037d58b470d976dc6f43aff7790 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c4cd0bf3767a4024ba327faf3015304c in term 0.
I20260806 06:26:28.210428 23793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:28.207032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.130:33755 (local address 127.21.12.129:48179)
0806 06:26:28.207260 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0806 06:26:28.207270 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0806 06:26:28.207330 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:26:28.207461 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:26:28.207462 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:26:28.207464 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 06:26:28.207685 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:28.207688 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:28.208520 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:28.208521 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:28.209414 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:28.209417 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:28.210036 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:28.210037 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:28.210200 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:28.210205 (+ 5us) client_negotiation.cc:769] Sending connection context
0806 06:26:28.210264 (+ 59us) client_negotiation.cc:240] Negotiation successful
0806 06:26:28.210325 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":83,"threads_started":1}
I20260806 06:26:28.210614 23733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:28.207099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.129:48179 (local address 127.21.12.130:33755)
0806 06:26:28.207180 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0806 06:26:28.207183 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 06:26:28.207327 (+ 144us) server_negotiation.cc:407] Connection header received
0806 06:26:28.207391 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:26:28.207392 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:26:28.207402 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:26:28.207450 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 06:26:28.207828 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:28.208442 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:28.209505 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:28.209956 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:28.210015 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:28.210023 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0806 06:26:28.210502 (+ 479us) server_negotiation.cc:299] Negotiation successful
0806 06:26:28.210554 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260806 06:26:28.210632 23146 leader_election.cc:302] T a4b5f07ac5054118a33ed7247abc4a51 P c4cd0bf3767a4024ba327faf3015304c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 279a4037d58b470d976dc6f43aff7790, c4cd0bf3767a4024ba327faf3015304c; no voters:
I20260806 06:26:28.210757 23746 raft_consensus.cc:2781] T a4b5f07ac5054118a33ed7247abc4a51 P c4cd0bf3767a4024ba327faf3015304c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 06:26:28.210803 23746 raft_consensus.cc:482] T a4b5f07ac5054118a33ed7247abc4a51 P c4cd0bf3767a4024ba327faf3015304c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 06:26:28.210829 23746 raft_consensus.cc:3037] T a4b5f07ac5054118a33ed7247abc4a51 P c4cd0bf3767a4024ba327faf3015304c [term 0 FOLLOWER]: Advancing to term 1
I20260806 06:26:28.210803 23435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4b5f07ac5054118a33ed7247abc4a51" candidate_uuid: "c4cd0bf3767a4024ba327faf3015304c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bdd4da71739a455c9495a8887b22fa09" is_pre_election: true
I20260806 06:26:28.210907 23435 raft_consensus.cc:2454] T a4b5f07ac5054118a33ed7247abc4a51 P bdd4da71739a455c9495a8887b22fa09 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c4cd0bf3767a4024ba327faf3015304c in term 0.
I20260806 06:26:28.211696 23746 raft_consensus.cc:504] T a4b5f07ac5054118a33ed7247abc4a51 P c4cd0bf3767a4024ba327faf3015304c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4cd0bf3767a4024ba327faf3015304c" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 33499 } } peers { permanent_uuid: "bdd4da71739a455c9495a8887b22fa09" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 33755 } } peers { permanent_uuid: "279a4037d58b470d976dc6f43aff7790" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 39473 } }
I20260806 06:26:28.211817 23746 leader_election.cc:288] T a4b5f07ac5054118a33ed7247abc4a51 P c4cd0bf3767a4024ba327faf3015304c [CANDIDATE]: Term 1 election: Requested vote from peers bdd4da71739a455c9495a8887b22fa09 (127.21.12.130:33755), 279a4037d58b470d976dc6f43aff7790 (127.21.12.131:39473)
I20260806 06:26:28.211979 23435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4b5f07ac5054118a33ed7247abc4a51" candidate_uuid: "c4cd0bf3767a4024ba327faf3015304c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bdd4da71739a455c9495a8887b22fa09"
I20260806 06:26:28.212085 23435 raft_consensus.cc:3037] T a4b5f07ac5054118a33ed7247abc4a51 P bdd4da71739a455c9495a8887b22fa09 [term 0 FOLLOWER]: Advancing to term 1
I20260806 06:26:28.212081 23643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4b5f07ac5054118a33ed7247abc4a51" candidate_uuid: "c4cd0bf3767a4024ba327faf3015304c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "279a4037d58b470d976dc6f43aff7790"
I20260806 06:26:28.212150 23643 raft_consensus.cc:3037] T a4b5f07ac5054118a33ed7247abc4a51 P 279a4037d58b470d976dc6f43aff7790 [term 0 FOLLOWER]: Advancing to term 1
I20260806 06:26:28.212934 23435 raft_consensus.cc:2454] T a4b5f07ac5054118a33ed7247abc4a51 P bdd4da71739a455c9495a8887b22fa09 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c4cd0bf3767a4024ba327faf3015304c in term 1.
I20260806 06:26:28.213155 23147 leader_election.cc:302] T a4b5f07ac5054118a33ed7247abc4a51 P c4cd0bf3767a4024ba327faf3015304c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bdd4da71739a455c9495a8887b22fa09, c4cd0bf3767a4024ba327faf3015304c; no voters:
I20260806 06:26:28.213212 23643 raft_consensus.cc:2454] T a4b5f07ac5054118a33ed7247abc4a51 P 279a4037d58b470d976dc6f43aff7790 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c4cd0bf3767a4024ba327faf3015304c in term 1.
I20260806 06:26:28.213246 23746 raft_consensus.cc:2781] T a4b5f07ac5054118a33ed7247abc4a51 P c4cd0bf3767a4024ba327faf3015304c [term 1 FOLLOWER]: Leader election won for term 1
I20260806 06:26:28.213340 23746 raft_consensus.cc:686] T a4b5f07ac5054118a33ed7247abc4a51 P c4cd0bf3767a4024ba327faf3015304c [term 1 LEADER]: Becoming Leader. State: Replica: c4cd0bf3767a4024ba327faf3015304c, State: Running, Role: LEADER
I20260806 06:26:28.213408 23746 consensus_queue.cc:237] T a4b5f07ac5054118a33ed7247abc4a51 P c4cd0bf3767a4024ba327faf3015304c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4cd0bf3767a4024ba327faf3015304c" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 33499 } } peers { permanent_uuid: "bdd4da71739a455c9495a8887b22fa09" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 33755 } } peers { permanent_uuid: "279a4037d58b470d976dc6f43aff7790" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 39473 } }
I20260806 06:26:28.213784 23794 tablet_replica.cc:439] T a4b5f07ac5054118a33ed7247abc4a51 P c4cd0bf3767a4024ba327faf3015304c: TxnStatusTablet state changed. Reason: New leader c4cd0bf3767a4024ba327faf3015304c. Latest consensus state: current_term: 1 leader_uuid: "c4cd0bf3767a4024ba327faf3015304c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4cd0bf3767a4024ba327faf3015304c" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 33499 } } peers { permanent_uuid: "bdd4da71739a455c9495a8887b22fa09" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 33755 } } peers { permanent_uuid: "279a4037d58b470d976dc6f43aff7790" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 39473 } } }
I20260806 06:26:28.213872 23794 tablet_replica.cc:442] T a4b5f07ac5054118a33ed7247abc4a51 P c4cd0bf3767a4024ba327faf3015304c: This TxnStatusTablet replica's current role is: LEADER
I20260806 06:26:28.214048 23796 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260806 06:26:28.214157 23796 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260806 06:26:28.214169 23031 catalog_manager.cc:5495] T a4b5f07ac5054118a33ed7247abc4a51 P c4cd0bf3767a4024ba327faf3015304c reported cstate change: term changed from 0 to 1, leader changed from <none> to c4cd0bf3767a4024ba327faf3015304c (127.21.12.129). New cstate: current_term: 1 leader_uuid: "c4cd0bf3767a4024ba327faf3015304c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4cd0bf3767a4024ba327faf3015304c" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 33499 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bdd4da71739a455c9495a8887b22fa09" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 33755 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "279a4037d58b470d976dc6f43aff7790" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 39473 } health_report { overall_health: UNKNOWN } } }
I20260806 06:26:28.235894 23735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:28.232966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40320 (local address 127.21.12.129:33499)
0806 06:26:28.233019 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0806 06:26:28.233021 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:26:28.233072 (+ 51us) server_negotiation.cc:407] Connection header received
0806 06:26:28.233160 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:26:28.233161 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:26:28.233182 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:26:28.233247 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 06:26:28.233419 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:28.234207 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:28.234990 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:28.235290 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:28.235358 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:28.235542 (+ 184us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 06:26:28.235689 (+ 147us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 06:26:28.235762 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0806 06:26:28.235795 (+ 33us) server_negotiation.cc:299] Negotiation successful
0806 06:26:28.235814 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260806 06:26:28.235919 23784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:28.232914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.129:33499 (local address 127.0.0.1:40320)
0806 06:26:28.233018 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0806 06:26:28.233029 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0806 06:26:28.233084 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:26:28.233290 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:26:28.233291 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:26:28.233293 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 06:26:28.233344 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:28.233346 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:28.234320 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:28.234321 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:28.234907 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:28.234909 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:28.235376 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:28.235377 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:28.235438 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:28.235453 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 06:26:28.235753 (+ 300us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 06:26:28.235762 (+ 9us) client_negotiation.cc:769] Sending connection context
0806 06:26:28.235790 (+ 28us) client_negotiation.cc:240] Negotiation successful
0806 06:26:28.235835 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260806 06:26:28.237317 23643 raft_consensus.cc:1261] T a4b5f07ac5054118a33ed7247abc4a51 P 279a4037d58b470d976dc6f43aff7790 [term 1 FOLLOWER]: Refusing update from remote peer c4cd0bf3767a4024ba327faf3015304c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 06:26:28.237452 23435 raft_consensus.cc:1261] T a4b5f07ac5054118a33ed7247abc4a51 P bdd4da71739a455c9495a8887b22fa09 [term 1 FOLLOWER]: Refusing update from remote peer c4cd0bf3767a4024ba327faf3015304c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 06:26:28.237569 23794 consensus_queue.cc:1035] T a4b5f07ac5054118a33ed7247abc4a51 P c4cd0bf3767a4024ba327faf3015304c [LEADER]: Connected to new peer: Peer: permanent_uuid: "279a4037d58b470d976dc6f43aff7790" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 39473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 06:26:28.237701 23746 consensus_queue.cc:1035] T a4b5f07ac5054118a33ed7247abc4a51 P c4cd0bf3767a4024ba327faf3015304c [LEADER]: Connected to new peer: Peer: permanent_uuid: "bdd4da71739a455c9495a8887b22fa09" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 33755 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 06:26:28.238234 23745 tablet_replica.cc:439] T a4b5f07ac5054118a33ed7247abc4a51 P 279a4037d58b470d976dc6f43aff7790: TxnStatusTablet state changed. Reason: New leader c4cd0bf3767a4024ba327faf3015304c. Latest consensus state: current_term: 1 leader_uuid: "c4cd0bf3767a4024ba327faf3015304c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4cd0bf3767a4024ba327faf3015304c" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 33499 } } peers { permanent_uuid: "bdd4da71739a455c9495a8887b22fa09" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 33755 } } peers { permanent_uuid: "279a4037d58b470d976dc6f43aff7790" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 39473 } } }
I20260806 06:26:28.238354 23745 tablet_replica.cc:442] T a4b5f07ac5054118a33ed7247abc4a51 P 279a4037d58b470d976dc6f43aff7790: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 06:26:28.238535 23748 tablet_replica.cc:439] T a4b5f07ac5054118a33ed7247abc4a51 P bdd4da71739a455c9495a8887b22fa09: TxnStatusTablet state changed. Reason: New leader c4cd0bf3767a4024ba327faf3015304c. Latest consensus state: current_term: 1 leader_uuid: "c4cd0bf3767a4024ba327faf3015304c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4cd0bf3767a4024ba327faf3015304c" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 33499 } } peers { permanent_uuid: "bdd4da71739a455c9495a8887b22fa09" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 33755 } } peers { permanent_uuid: "279a4037d58b470d976dc6f43aff7790" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 39473 } } }
I20260806 06:26:28.238623 23748 tablet_replica.cc:442] T a4b5f07ac5054118a33ed7247abc4a51 P bdd4da71739a455c9495a8887b22fa09: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 06:26:28.239179 23794 tablet_replica.cc:439] T a4b5f07ac5054118a33ed7247abc4a51 P c4cd0bf3767a4024ba327faf3015304c: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c4cd0bf3767a4024ba327faf3015304c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4cd0bf3767a4024ba327faf3015304c" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 33499 } } peers { permanent_uuid: "bdd4da71739a455c9495a8887b22fa09" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 33755 } } peers { permanent_uuid: "279a4037d58b470d976dc6f43aff7790" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 39473 } } }
I20260806 06:26:28.239257 23794 tablet_replica.cc:442] T a4b5f07ac5054118a33ed7247abc4a51 P c4cd0bf3767a4024ba327faf3015304c: This TxnStatusTablet replica's current role is: LEADER
I20260806 06:26:28.239362 23752 mvcc.cc:205] Tried to move back new op lower bound from 7315446121417961472 to 7315446121322815488. Current Snapshot: MvccSnapshot[applied={T|T < 7315446121417961472 or (T in {7315446121417961472})}]
I20260806 06:26:28.239668 23748 tablet_replica.cc:439] T a4b5f07ac5054118a33ed7247abc4a51 P bdd4da71739a455c9495a8887b22fa09: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c4cd0bf3767a4024ba327faf3015304c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4cd0bf3767a4024ba327faf3015304c" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 33499 } } peers { permanent_uuid: "bdd4da71739a455c9495a8887b22fa09" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 33755 } } peers { permanent_uuid: "279a4037d58b470d976dc6f43aff7790" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 39473 } } }
I20260806 06:26:28.239750 23748 tablet_replica.cc:442] T a4b5f07ac5054118a33ed7247abc4a51 P bdd4da71739a455c9495a8887b22fa09: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 06:26:28.240671 23794 tablet_replica.cc:439] T a4b5f07ac5054118a33ed7247abc4a51 P c4cd0bf3767a4024ba327faf3015304c: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c4cd0bf3767a4024ba327faf3015304c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4cd0bf3767a4024ba327faf3015304c" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 33499 } } peers { permanent_uuid: "bdd4da71739a455c9495a8887b22fa09" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 33755 } } peers { permanent_uuid: "279a4037d58b470d976dc6f43aff7790" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 39473 } } }
I20260806 06:26:28.240772 23794 tablet_replica.cc:442] T a4b5f07ac5054118a33ed7247abc4a51 P c4cd0bf3767a4024ba327faf3015304c: This TxnStatusTablet replica's current role is: LEADER
I20260806 06:26:28.241315 23745 tablet_replica.cc:439] T a4b5f07ac5054118a33ed7247abc4a51 P 279a4037d58b470d976dc6f43aff7790: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c4cd0bf3767a4024ba327faf3015304c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4cd0bf3767a4024ba327faf3015304c" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 33499 } } peers { permanent_uuid: "bdd4da71739a455c9495a8887b22fa09" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 33755 } } peers { permanent_uuid: "279a4037d58b470d976dc6f43aff7790" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 39473 } } }
I20260806 06:26:28.241401 23745 tablet_replica.cc:442] T a4b5f07ac5054118a33ed7247abc4a51 P 279a4037d58b470d976dc6f43aff7790: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 06:26:28.244241 23734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:28.241543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56930 (local address 127.21.12.131:39473)
0806 06:26:28.241580 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0806 06:26:28.241582 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:26:28.241589 (+ 7us) server_negotiation.cc:407] Connection header received
0806 06:26:28.241623 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:26:28.241624 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:26:28.241644 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:26:28.241702 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 06:26:28.241941 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:28.242711 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:28.243442 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:28.243787 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:28.243857 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:28.243949 (+ 92us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 06:26:28.244066 (+ 117us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 06:26:28.244140 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0806 06:26:28.244155 (+ 15us) server_negotiation.cc:299] Negotiation successful
0806 06:26:28.244174 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260806 06:26:28.249531 23735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:28.247019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.131:35861 (local address 127.21.12.129:33499)
0806 06:26:28.247071 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0806 06:26:28.247073 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:26:28.247079 (+ 6us) server_negotiation.cc:407] Connection header received
0806 06:26:28.247096 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:26:28.247096 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:26:28.247103 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:26:28.247143 (+ 40us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 06:26:28.247367 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:28.247784 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:28.248872 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:28.249051 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:28.249102 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:28.249322 (+ 220us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 06:26:28.249346 (+ 24us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 06:26:28.249379 (+ 33us) server_negotiation.cc:1035] Waiting for connection context
0806 06:26:28.249456 (+ 77us) server_negotiation.cc:299] Negotiation successful
0806 06:26:28.249472 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260806 06:26:28.249580 23718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:28.246815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.129:33499 (local address 127.21.12.131:35861)
0806 06:26:28.246892 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0806 06:26:28.246904 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 06:26:28.246950 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:26:28.247165 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:26:28.247167 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:26:28.247168 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 06:26:28.247303 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:28.247305 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:28.247851 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:28.247852 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:28.248785 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:28.248788 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:28.249117 (+ 329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:28.249118 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:28.249223 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:28.249239 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 06:26:28.249399 (+ 160us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 06:26:28.249400 (+ 1us) client_negotiation.cc:769] Sending connection context
0806 06:26:28.249454 (+ 54us) client_negotiation.cc:240] Negotiation successful
0806 06:26:28.249502 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260806 06:26:28.252007 23435 raft_consensus.cc:1261] T c07e37b5f6ac41e2b784acffa978ed25 P bdd4da71739a455c9495a8887b22fa09 [term 1 FOLLOWER]: Refusing update from remote peer 279a4037d58b470d976dc6f43aff7790: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 06:26:28.252008 23240 raft_consensus.cc:1261] T c07e37b5f6ac41e2b784acffa978ed25 P c4cd0bf3767a4024ba327faf3015304c [term 1 FOLLOWER]: Refusing update from remote peer 279a4037d58b470d976dc6f43aff7790: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 06:26:28.252255 23745 consensus_queue.cc:1035] T c07e37b5f6ac41e2b784acffa978ed25 P 279a4037d58b470d976dc6f43aff7790 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4cd0bf3767a4024ba327faf3015304c" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 33499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 06:26:28.252460 23775 consensus_queue.cc:1035] T c07e37b5f6ac41e2b784acffa978ed25 P 279a4037d58b470d976dc6f43aff7790 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bdd4da71739a455c9495a8887b22fa09" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 33755 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 06:26:28.262940 23734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:28.259957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.129:55955 (local address 127.21.12.131:39473)
0806 06:26:28.260037 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0806 06:26:28.260039 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:26:28.260208 (+ 169us) server_negotiation.cc:407] Connection header received
0806 06:26:28.260309 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:26:28.260310 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:26:28.260322 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:26:28.260369 (+ 47us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 06:26:28.260559 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:28.261003 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:28.261895 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:28.262191 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:28.262237 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:28.262662 (+ 425us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 06:26:28.262696 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 06:26:28.262725 (+ 29us) server_negotiation.cc:1035] Waiting for connection context
0806 06:26:28.262828 (+ 103us) server_negotiation.cc:299] Negotiation successful
0806 06:26:28.262850 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260806 06:26:28.263248 23809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:28.259859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.131:39473 (local address 127.21.12.129:55955)
0806 06:26:28.260148 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0806 06:26:28.260159 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0806 06:26:28.260233 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:26:28.260409 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:26:28.260410 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:26:28.260411 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 06:26:28.260492 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:28.260494 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:28.261070 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:28.261071 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:28.261786 (+ 715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:28.261789 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:28.262371 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:28.262372 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:28.262531 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:28.262550 (+ 19us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 06:26:28.262764 (+ 214us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 06:26:28.262766 (+ 2us) client_negotiation.cc:769] Sending connection context
0806 06:26:28.263070 (+ 304us) client_negotiation.cc:240] Negotiation successful
0806 06:26:28.263113 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":123,"threads_started":1}
I20260806 06:26:28.271065 23031 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33586:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 06:26:28.271153 23031 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33586:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 06:26:28.271677 23031 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 31c5e8e609c44afa8e0f2b07d741d0af: Sending DeleteTablet for 3 replicas of tablet c07e37b5f6ac41e2b784acffa978ed25
I20260806 06:26:28.271879 23214 tablet_service.cc:1514] Processing DeleteTablet for tablet c07e37b5f6ac41e2b784acffa978ed25 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 06:26:28 UTC) from {username='kudu', principal='kudu/127.21.12.190@KRBTEST.COM'} at 127.0.0.1:40312
I20260806 06:26:28.271979 23766 tablet_replica.cc:330] T c07e37b5f6ac41e2b784acffa978ed25 P c4cd0bf3767a4024ba327faf3015304c: stopping tablet replica
I20260806 06:26:28.272032 23766 raft_consensus.cc:2229] T c07e37b5f6ac41e2b784acffa978ed25 P c4cd0bf3767a4024ba327faf3015304c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 06:26:28.272014 23411 tablet_service.cc:1514] Processing DeleteTablet for tablet c07e37b5f6ac41e2b784acffa978ed25 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 06:26:28 UTC) from {username='kudu', principal='kudu/127.21.12.190@KRBTEST.COM'} at 127.0.0.1:45980
I20260806 06:26:28.272073 23766 raft_consensus.cc:2258] T c07e37b5f6ac41e2b784acffa978ed25 P c4cd0bf3767a4024ba327faf3015304c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 06:26:28.272092 23764 tablet_replica.cc:330] T c07e37b5f6ac41e2b784acffa978ed25 P bdd4da71739a455c9495a8887b22fa09: stopping tablet replica
I20260806 06:26:28.272141 23764 raft_consensus.cc:2229] T c07e37b5f6ac41e2b784acffa978ed25 P bdd4da71739a455c9495a8887b22fa09 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 06:26:28.272171 23764 raft_consensus.cc:2258] T c07e37b5f6ac41e2b784acffa978ed25 P bdd4da71739a455c9495a8887b22fa09 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 06:26:28.272233 23619 tablet_service.cc:1514] Processing DeleteTablet for tablet c07e37b5f6ac41e2b784acffa978ed25 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 06:26:28 UTC) from {username='kudu', principal='kudu/127.21.12.190@KRBTEST.COM'} at 127.0.0.1:56918
I20260806 06:26:28.272312 23765 tablet_replica.cc:330] T c07e37b5f6ac41e2b784acffa978ed25 P 279a4037d58b470d976dc6f43aff7790: stopping tablet replica
I20260806 06:26:28.272357 23765 raft_consensus.cc:2229] T c07e37b5f6ac41e2b784acffa978ed25 P 279a4037d58b470d976dc6f43aff7790 [term 1 LEADER]: Raft consensus shutting down.
I20260806 06:26:28.272433 23765 raft_consensus.cc:2258] T c07e37b5f6ac41e2b784acffa978ed25 P 279a4037d58b470d976dc6f43aff7790 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 06:26:28.272449 23766 ts_tablet_manager.cc:1905] T c07e37b5f6ac41e2b784acffa978ed25 P c4cd0bf3767a4024ba327faf3015304c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 06:26:28.273283 23764 ts_tablet_manager.cc:1905] T c07e37b5f6ac41e2b784acffa978ed25 P bdd4da71739a455c9495a8887b22fa09: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 06:26:28.274400 23766 ts_tablet_manager.cc:1918] T c07e37b5f6ac41e2b784acffa978ed25 P c4cd0bf3767a4024ba327faf3015304c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 06:26:28.274462 23766 log.cc:1196] T c07e37b5f6ac41e2b784acffa978ed25 P c4cd0bf3767a4024ba327faf3015304c: Deleting WAL directory at /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785997581049705-21554-0/minicluster-data/ts-0/wal/wals/c07e37b5f6ac41e2b784acffa978ed25
I20260806 06:26:28.274561 23764 ts_tablet_manager.cc:1918] T c07e37b5f6ac41e2b784acffa978ed25 P bdd4da71739a455c9495a8887b22fa09: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 06:26:28.274616 23764 log.cc:1196] T c07e37b5f6ac41e2b784acffa978ed25 P bdd4da71739a455c9495a8887b22fa09: Deleting WAL directory at /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785997581049705-21554-0/minicluster-data/ts-1/wal/wals/c07e37b5f6ac41e2b784acffa978ed25
I20260806 06:26:28.274679 23766 ts_tablet_manager.cc:1939] T c07e37b5f6ac41e2b784acffa978ed25 P c4cd0bf3767a4024ba327faf3015304c: Deleting consensus metadata
I20260806 06:26:28.274825 23764 ts_tablet_manager.cc:1939] T c07e37b5f6ac41e2b784acffa978ed25 P bdd4da71739a455c9495a8887b22fa09: Deleting consensus metadata
I20260806 06:26:28.275041 23765 ts_tablet_manager.cc:1905] T c07e37b5f6ac41e2b784acffa978ed25 P 279a4037d58b470d976dc6f43aff7790: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 06:26:28.275120 23014 catalog_manager.cc:4841] TS bdd4da71739a455c9495a8887b22fa09 (127.21.12.130:33755): tablet c07e37b5f6ac41e2b784acffa978ed25 (table test-table [id=9e140931887845f5b18bee384d60827d]) successfully deleted
I20260806 06:26:28.275295 23012 catalog_manager.cc:4841] TS c4cd0bf3767a4024ba327faf3015304c (127.21.12.129:33499): tablet c07e37b5f6ac41e2b784acffa978ed25 (table test-table [id=9e140931887845f5b18bee384d60827d]) successfully deleted
I20260806 06:26:28.276407 23765 ts_tablet_manager.cc:1918] T c07e37b5f6ac41e2b784acffa978ed25 P 279a4037d58b470d976dc6f43aff7790: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 06:26:28.276443 23765 log.cc:1196] T c07e37b5f6ac41e2b784acffa978ed25 P 279a4037d58b470d976dc6f43aff7790: Deleting WAL directory at /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785997581049705-21554-0/minicluster-data/ts-2/wal/wals/c07e37b5f6ac41e2b784acffa978ed25
I20260806 06:26:28.276573 23765 ts_tablet_manager.cc:1939] T c07e37b5f6ac41e2b784acffa978ed25 P 279a4037d58b470d976dc6f43aff7790: Deleting consensus metadata
I20260806 06:26:28.276952 23012 catalog_manager.cc:4841] TS 279a4037d58b470d976dc6f43aff7790 (127.21.12.131:39473): tablet c07e37b5f6ac41e2b784acffa978ed25 (table test-table [id=9e140931887845f5b18bee384d60827d]) successfully deleted
Aug 06 06:26:28 dist-test-slave-4tdj krb5kdc[22970](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997587, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.21.12.129@KRBTEST.COM
I20260806 06:26:28.286113 23735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:28.273309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40330 (local address 127.21.12.129:33499)
0806 06:26:28.273344 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0806 06:26:28.273346 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:26:28.278464 (+ 5118us) server_negotiation.cc:407] Connection header received
0806 06:26:28.278523 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:26:28.278524 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:26:28.278535 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:26:28.278579 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:26:28.279130 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:28.279596 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:28.280373 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:28.280575 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:28.280605 (+ 30us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:28.283494 (+ 2889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:26:28.283504 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:26:28.283504 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 06:26:28.283511 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:26:28.285076 (+ 1565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:26:28.285398 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:26:28.285401 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:26:28.285401 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:26:28.285452 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:26:28.285620 (+ 168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:26:28.285620 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:26:28.285621 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:26:28.285748 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:26:28.285887 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0806 06:26:28.285948 (+ 61us) server_negotiation.cc:299] Negotiation successful
0806 06:26:28.286027 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
W20260806 06:26:28.286391 23293 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40330
I20260806 06:26:28.292670 23096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:28.287382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33598 (local address 127.21.12.190:41067)
0806 06:26:28.287423 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0806 06:26:28.287424 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0806 06:26:28.287482 (+ 58us) server_negotiation.cc:407] Connection header received
0806 06:26:28.287547 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:26:28.287548 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:26:28.287557 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:26:28.287598 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:26:28.288064 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:28.288552 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:28.289171 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:28.289370 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:28.289422 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:28.290479 (+ 1057us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:26:28.290480 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:26:28.290481 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 06:26:28.290487 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:26:28.291692 (+ 1205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:26:28.292003 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:26:28.292004 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:26:28.292004 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:26:28.292060 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:26:28.292217 (+ 157us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:26:28.292218 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:26:28.292219 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:26:28.292323 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:26:28.292406 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0806 06:26:28.292514 (+ 108us) server_negotiation.cc:299] Negotiation successful
0806 06:26:28.292584 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
W20260806 06:26:28.292871 23031 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33598
I20260806 06:26:28.293524 21554 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHhLhT/build/release/bin/kudu with pid 23124
I20260806 06:26:28.306089 21554 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHhLhT/build/release/bin/kudu with pid 23317
I20260806 06:26:28.325444 21554 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHhLhT/build/release/bin/kudu with pid 23519
I20260806 06:26:28.334003 21554 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHhLhT/build/release/bin/kudu with pid 22984
2026-08-06T06:26:28Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2265 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785997581049705-21554-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 06:26:28 dist-test-slave-4tdj krb5kdc[23821](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 06:26:28 dist-test-slave-4tdj krb5kdc[23821](info): set up 2 sockets
Aug 06 06:26:28 dist-test-slave-4tdj krb5kdc[23821](info): commencing operation
krb5kdc: starting...
W20260806 06:26:29.465240 21554 mini_kdc.cc:121] Time spent starting KDC: real 1.107s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785997581049705-21554-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785997581049705-21554-0/krb5kdc/test-user.keytab.
Aug 06 06:26:29 dist-test-slave-4tdj krb5kdc[23821](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997589, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T06:26:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T06:26:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.12.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.12.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.12.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.190.keytab.
Entry for principal kudu/127.21.12.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.190.keytab.
I20260806 06:26:29.578640 21554 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHhLhT/build/release/bin/kudu
/tmp/dist-test-taskFHhLhT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785997581049705-21554-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785997581049705-21554-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785997581049705-21554-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785997581049705-21554-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.12.190:43693
--webserver_interface=127.21.12.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.190.keytab
--principal=kudu/127.21.12.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.148:38149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.12.190:43693
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785997581049705-21554-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785997581049705-21554-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785997581049705-21554-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 06:26:29.657146 23842 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 06:26:29.657341 23842 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 06:26:29.657371 23842 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 06:26:29.658504 23842 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 06:26:29.658540 23842 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260806 06:26:29.658557 23842 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 06:26:29.658576 23842 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 06:26:29.658591 23842 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 06:26:29.658607 23842 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 06:26:29.660064 23842 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.148:38149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785997581049705-21554-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785997581049705-21554-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.12.190:43693
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.190.keytab
--principal=kudu/127.21.12.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.12.190:43693
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785997581049705-21554-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.12.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23842
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785997581049705-21554-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 05:11:02 UTC on ccycloud.quasar-hvzbbz.root.comops.site
build id 4569617
I20260806 06:26:29.660295 23842 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 06:26:29.660482 23842 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 06:26:29.665154 23849 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:26:29.665201 23848 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:26:29.665249 23842 server_base.cc:1034] running on GCE node
W20260806 06:26:29.665177 23852 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:26:29.665513 23842 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 06:26:29.665769 23842 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 06:26:29.666931 23842 hybrid_clock.cc:648] HybridClock initialized: now 1785997589666917 us; error 29 us; skew 500 ppm
Aug 06 06:26:29 dist-test-slave-4tdj krb5kdc[23821](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997589, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 06:26:29.669378 23842 init.cc:377] Logged in from keytab as kudu/127.21.12.190@KRBTEST.COM (short username kudu)
I20260806 06:26:29.669809 23842 webserver.cc:460] Webserver started at http://127.21.12.190:36275/ using document root <none> and password file <none>
I20260806 06:26:29.670018 23842 fs_manager.cc:359] Metadata directory not provided
I20260806 06:26:29.670082 23842 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 06:26:29.670207 23842 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 06:26:29.671312 23842 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785997581049705-21554-0/minicluster-data/master-0/data/instance:
uuid: "b3579394671149a89ff9b5c9f4b9854e"
format_stamp: "Formatted at 2026-08-06 06:26:29 on dist-test-slave-4tdj"
server_key: "eb22d60142e8402c8b1aa415bf7162fa"
server_key_iv: "6a918ab6a58999d46a0a387d7ec59a41"
server_key_version: "encryptionkey@0"
I20260806 06:26:29.671650 23842 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785997581049705-21554-0/minicluster-data/master-0/wal/instance:
uuid: "b3579394671149a89ff9b5c9f4b9854e"
format_stamp: "Formatted at 2026-08-06 06:26:29 on dist-test-slave-4tdj"
server_key: "eb22d60142e8402c8b1aa415bf7162fa"
server_key_iv: "6a918ab6a58999d46a0a387d7ec59a41"
server_key_version: "encryptionkey@0"
I20260806 06:26:29.672978 23842 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260806 06:26:29.673673 23860 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 06:26:29.673835 23842 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260806 06:26:29.673920 23842 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785997581049705-21554-0/minicluster-data/master-0/data,/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785997581049705-21554-0/minicluster-data/master-0/wal
uuid: "b3579394671149a89ff9b5c9f4b9854e"
format_stamp: "Formatted at 2026-08-06 06:26:29 on dist-test-slave-4tdj"
server_key: "eb22d60142e8402c8b1aa415bf7162fa"
server_key_iv: "6a918ab6a58999d46a0a387d7ec59a41"
server_key_version: "encryptionkey@0"
I20260806 06:26:29.673978 23842 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785997581049705-21554-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785997581049705-21554-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785997581049705-21554-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 06:26:29.681283 23842 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 06:26:29.683090 23842 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 06:26:29.683215 23842 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 06:26:29.687697 23842 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.190:43693
I20260806 06:26:29.687747 23943 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.190:43693 every 8 connection(s)
I20260806 06:26:29.688046 23842 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785997581049705-21554-0/minicluster-data/master-0/data/info.pb
I20260806 06:26:29.688705 23944 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 06:26:29.690959 23944 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3579394671149a89ff9b5c9f4b9854e: Bootstrap starting.
I20260806 06:26:29.691525 23944 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b3579394671149a89ff9b5c9f4b9854e: Neither blocks nor log segments found. Creating new log.
I20260806 06:26:29.691787 23944 log.cc:824] T 00000000000000000000000000000000 P b3579394671149a89ff9b5c9f4b9854e: Log is configured to *not* fsync() on all Append() calls
I20260806 06:26:29.692411 23944 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3579394671149a89ff9b5c9f4b9854e: No bootstrap required, opened a new log
I20260806 06:26:29.692871 21554 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHhLhT/build/release/bin/kudu as pid 23842
I20260806 06:26:29.692950 21554 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785997581049705-21554-0/minicluster-data/master-0/wal/instance
I20260806 06:26:29.693101 21554 external_mini_cluster.cc:1428] Setting key c108fc2b68c26a06a1308e3f955b48d0
I20260806 06:26:29.694715 23944 raft_consensus.cc:348] T 00000000000000000000000000000000 P b3579394671149a89ff9b5c9f4b9854e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3579394671149a89ff9b5c9f4b9854e" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 43693 } }
I20260806 06:26:29.694865 23944 raft_consensus.cc:374] T 00000000000000000000000000000000 P b3579394671149a89ff9b5c9f4b9854e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 06:26:29.694912 23944 raft_consensus.cc:729] T 00000000000000000000000000000000 P b3579394671149a89ff9b5c9f4b9854e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3579394671149a89ff9b5c9f4b9854e, State: Initialized, Role: FOLLOWER
I20260806 06:26:29.695055 23944 consensus_queue.cc:260] T 00000000000000000000000000000000 P b3579394671149a89ff9b5c9f4b9854e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3579394671149a89ff9b5c9f4b9854e" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 43693 } }
I20260806 06:26:29.695139 23944 raft_consensus.cc:388] T 00000000000000000000000000000000 P b3579394671149a89ff9b5c9f4b9854e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 06:26:29.695176 23944 raft_consensus.cc:482] T 00000000000000000000000000000000 P b3579394671149a89ff9b5c9f4b9854e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 06:26:29.695225 23944 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b3579394671149a89ff9b5c9f4b9854e [term 0 FOLLOWER]: Advancing to term 1
I20260806 06:26:29.695832 23944 raft_consensus.cc:504] T 00000000000000000000000000000000 P b3579394671149a89ff9b5c9f4b9854e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3579394671149a89ff9b5c9f4b9854e" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 43693 } }
I20260806 06:26:29.695924 23944 leader_election.cc:302] T 00000000000000000000000000000000 P b3579394671149a89ff9b5c9f4b9854e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b3579394671149a89ff9b5c9f4b9854e; no voters:
I20260806 06:26:29.696096 23944 leader_election.cc:288] T 00000000000000000000000000000000 P b3579394671149a89ff9b5c9f4b9854e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 06:26:29.696127 23956 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b3579394671149a89ff9b5c9f4b9854e [term 1 FOLLOWER]: Leader election won for term 1
I20260806 06:26:29.696239 23956 raft_consensus.cc:686] T 00000000000000000000000000000000 P b3579394671149a89ff9b5c9f4b9854e [term 1 LEADER]: Becoming Leader. State: Replica: b3579394671149a89ff9b5c9f4b9854e, State: Running, Role: LEADER
I20260806 06:26:29.696319 23956 consensus_queue.cc:237] T 00000000000000000000000000000000 P b3579394671149a89ff9b5c9f4b9854e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3579394671149a89ff9b5c9f4b9854e" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 43693 } }
I20260806 06:26:29.696353 23944 sys_catalog.cc:564] T 00000000000000000000000000000000 P b3579394671149a89ff9b5c9f4b9854e [sys.catalog]: configured and running, proceeding with master startup.
I20260806 06:26:29.696683 23957 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3579394671149a89ff9b5c9f4b9854e [sys.catalog]: SysCatalogTable state changed. Reason: New leader b3579394671149a89ff9b5c9f4b9854e. Latest consensus state: current_term: 1 leader_uuid: "b3579394671149a89ff9b5c9f4b9854e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3579394671149a89ff9b5c9f4b9854e" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 43693 } } }
I20260806 06:26:29.696758 23957 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3579394671149a89ff9b5c9f4b9854e [sys.catalog]: This master's current role is: LEADER
I20260806 06:26:29.696970 23960 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 06:26:29.697484 23958 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3579394671149a89ff9b5c9f4b9854e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b3579394671149a89ff9b5c9f4b9854e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3579394671149a89ff9b5c9f4b9854e" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 43693 } } }
I20260806 06:26:29.697626 23960 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 06:26:29.697631 23958 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3579394671149a89ff9b5c9f4b9854e [sys.catalog]: This master's current role is: LEADER
I20260806 06:26:29.699268 23960 catalog_manager.cc:1347] Generated new cluster ID: b286833221bc43d1ac3c1f6d83a549dd
I20260806 06:26:29.699312 23960 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 06 06:26:29 dist-test-slave-4tdj krb5kdc[23821](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997589, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.12.190@KRBTEST.COM
I20260806 06:26:29.706036 23955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:29.693593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47530 (local address 127.21.12.190:43693)
0806 06:26:29.693736 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0806 06:26:29.693744 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 06:26:29.693768 (+ 24us) server_negotiation.cc:407] Connection header received
0806 06:26:29.694682 (+ 914us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:26:29.694689 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:26:29.694735 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:26:29.694845 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:26:29.695581 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:29.696736 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:29.697873 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:29.698394 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:29.698500 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:29.701630 (+ 3130us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:26:29.701634 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:26:29.701635 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 06:26:29.701646 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:26:29.704142 (+ 2496us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:26:29.704509 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:26:29.704512 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:26:29.704512 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:26:29.704621 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:26:29.704789 (+ 168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:26:29.704790 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:26:29.704790 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:26:29.705072 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:26:29.705201 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0806 06:26:29.705680 (+ 479us) server_negotiation.cc:299] Negotiation successful
0806 06:26:29.705823 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99,"thread_start_us":59,"threads_started":1}
I20260806 06:26:29.713511 23960 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 06:26:29.714008 23960 catalog_manager.cc:1504] Loading token signing keys...
I20260806 06:26:29.718639 23960 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b3579394671149a89ff9b5c9f4b9854e: Generated new TSK 0
I20260806 06:26:29.718803 23960 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.12.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.12.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.12.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.129.keytab.
Entry for principal kudu/127.21.12.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.129.keytab.
I20260806 06:26:29.746724 21554 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHhLhT/build/release/bin/kudu
/tmp/dist-test-taskFHhLhT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785997581049705-21554-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785997581049705-21554-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785997581049705-21554-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785997581049705-21554-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.12.129:0
--local_ip_for_outbound_sockets=127.21.12.129
--webserver_interface=127.21.12.129
--webserver_port=0
--tserver_master_addrs=127.21.12.190:43693
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.129.keytab
--principal=kudu/127.21.12.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.148:38149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785997581049705-21554-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785997581049705-21554-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785997581049705-21554-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 06:26:29.841738 23986 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 06:26:29.841939 23986 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 06:26:29.841971 23986 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 06:26:29.843120 23986 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 06:26:29.843158 23986 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 06:26:29.843184 23986 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260806 06:26:29.843199 23986 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 06:26:29.843222 23986 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.12.129
I20260806 06:26:29.844658 23986 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.148:38149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785997581049705-21554-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785997581049705-21554-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.129.keytab
--principal=kudu/127.21.12.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.12.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785997581049705-21554-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.12.129
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.21.12.190:43693
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23986
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.12.129
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785997581049705-21554-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 05:11:02 UTC on ccycloud.quasar-hvzbbz.root.comops.site
build id 4569617
I20260806 06:26:29.844882 23986 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 06:26:29.845098 23986 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 06:26:29.849838 23999 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:26:29.849825 23994 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:26:29.849825 23995 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:26:29.850085 23986 server_base.cc:1034] running on GCE node
I20260806 06:26:29.850271 23986 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 06:26:29.850510 23986 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 06:26:29.851639 23986 hybrid_clock.cc:648] HybridClock initialized: now 1785997589851630 us; error 29 us; skew 500 ppm
Aug 06 06:26:29 dist-test-slave-4tdj krb5kdc[23821](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997589, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 06:26:29.853953 23986 init.cc:377] Logged in from keytab as kudu/127.21.12.129@KRBTEST.COM (short username kudu)
I20260806 06:26:29.854379 23986 webserver.cc:460] Webserver started at http://127.21.12.129:43347/ using document root <none> and password file <none>
I20260806 06:26:29.854599 23986 fs_manager.cc:359] Metadata directory not provided
I20260806 06:26:29.854646 23986 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 06:26:29.854768 23986 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 06:26:29.855774 23986 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785997581049705-21554-0/minicluster-data/ts-0/data/instance:
uuid: "9447631f55b7404f9ab0232e0a1a2dc8"
format_stamp: "Formatted at 2026-08-06 06:26:29 on dist-test-slave-4tdj"
server_key: "3e069d524b049893a116a26dae6093cc"
server_key_iv: "3d13ffb255914be75d963428747e818b"
server_key_version: "encryptionkey@0"
I20260806 06:26:29.856093 23986 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785997581049705-21554-0/minicluster-data/ts-0/wal/instance:
uuid: "9447631f55b7404f9ab0232e0a1a2dc8"
format_stamp: "Formatted at 2026-08-06 06:26:29 on dist-test-slave-4tdj"
server_key: "3e069d524b049893a116a26dae6093cc"
server_key_iv: "3d13ffb255914be75d963428747e818b"
server_key_version: "encryptionkey@0"
I20260806 06:26:29.857333 23986 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260806 06:26:29.858054 24008 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 06:26:29.858244 23986 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260806 06:26:29.858356 23986 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785997581049705-21554-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785997581049705-21554-0/minicluster-data/ts-0/wal
uuid: "9447631f55b7404f9ab0232e0a1a2dc8"
format_stamp: "Formatted at 2026-08-06 06:26:29 on dist-test-slave-4tdj"
server_key: "3e069d524b049893a116a26dae6093cc"
server_key_iv: "3d13ffb255914be75d963428747e818b"
server_key_version: "encryptionkey@0"
I20260806 06:26:29.858430 23986 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785997581049705-21554-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785997581049705-21554-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785997581049705-21554-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 06:26:29.878839 23986 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 06:26:29.880999 23986 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 06:26:29.881114 23986 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 06:26:29.881650 23986 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 06:26:29.881693 23986 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 06:26:29.881722 23986 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 06:26:29.881738 23986 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 06:26:29 dist-test-slave-4tdj krb5kdc[23821](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997589, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.129@KRBTEST.COM for kudu/127.21.12.190@KRBTEST.COM
I20260806 06:26:29.888278 23986 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.129:46063
I20260806 06:26:29.888352 24180 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.129:46063 every 8 connection(s)
I20260806 06:26:29.888681 23986 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785997581049705-21554-0/minicluster-data/ts-0/data/info.pb
I20260806 06:26:29.891026 23955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:29.882125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.129:34535 (local address 127.21.12.190:43693)
0806 06:26:29.882193 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0806 06:26:29.882195 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:26:29.882424 (+ 229us) server_negotiation.cc:407] Connection header received
0806 06:26:29.882953 (+ 529us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:26:29.882955 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:26:29.882980 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:26:29.883054 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:26:29.883851 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:29.884599 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:29.885604 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:29.885912 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:29.885993 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:29.888439 (+ 2446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:26:29.888441 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:26:29.888441 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 06:26:29.888448 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:26:29.889802 (+ 1354us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:26:29.890188 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:26:29.890189 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:26:29.890189 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:26:29.890250 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:26:29.890522 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:26:29.890523 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:26:29.890523 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:26:29.890652 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:26:29.890700 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0806 06:26:29.890893 (+ 193us) server_negotiation.cc:299] Negotiation successful
0806 06:26:29.890964 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260806 06:26:29.891178 24020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:29.882094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:43693 (local address 127.21.12.129:34535)
0806 06:26:29.882271 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0806 06:26:29.882340 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0806 06:26:29.882837 (+ 497us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:26:29.883137 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:26:29.883141 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:26:29.883474 (+ 333us) client_negotiation.cc:189] Negotiated authn=SASL
0806 06:26:29.883747 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:29.883754 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:29.884713 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:29.884714 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:29.885536 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:29.885539 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:29.886012 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:29.886013 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:29.886154 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:29.886723 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 06:26:29.886731 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0806 06:26:29.888333 (+ 1602us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 06:26:29.889878 (+ 1545us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:26:29.889882 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:26:29.889883 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:26:29.890098 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:26:29.890337 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:26:29.890337 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:26:29.890338 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:26:29.890447 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:26:29.890697 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 06:26:29.890700 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 06:26:29.890808 (+ 108us) client_negotiation.cc:769] Sending connection context
0806 06:26:29.890879 (+ 71us) client_negotiation.cc:240] Negotiation successful
0806 06:26:29.890997 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":79,"threads_started":1}
I20260806 06:26:29.891631 24181 heartbeater.cc:347] Connected to a master server at 127.21.12.190:43693
I20260806 06:26:29.891718 24181 heartbeater.cc:464] Registering TS with master...
I20260806 06:26:29.891928 24181 heartbeater.cc:511] Master 127.21.12.190:43693 requested a full tablet report, sending...
I20260806 06:26:29.891980 21554 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHhLhT/build/release/bin/kudu as pid 23986
I20260806 06:26:29.892074 21554 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785997581049705-21554-0/minicluster-data/ts-0/wal/instance
I20260806 06:26:29.892274 21554 external_mini_cluster.cc:1428] Setting key 142cb778612eb2b98b3c8847844ab9e6
I20260806 06:26:29.892503 23882 ts_manager.cc:194] Registered new tserver with Master: 9447631f55b7404f9ab0232e0a1a2dc8 (127.21.12.129:46063)
I20260806 06:26:29.893347 23882 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.12.129@KRBTEST.COM'} at 127.21.12.129:34535
I20260806 06:26:29.898805 24187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:29.892852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:43693 (local address 127.21.12.129:38635)
0806 06:26:29.893037 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0806 06:26:29.893047 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0806 06:26:29.893101 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:26:29.893214 (+ 113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:26:29.893215 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:26:29.893378 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0806 06:26:29.893438 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:29.893439 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:29.894344 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:29.894345 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:29.895074 (+ 729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:29.895077 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:29.895401 (+ 324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:29.895402 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:29.895470 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:29.895849 (+ 379us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 06:26:29.895852 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0806 06:26:29.896214 (+ 362us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 06:26:29.897803 (+ 1589us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:26:29.897804 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:26:29.897804 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:26:29.897989 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:26:29.898195 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:26:29.898196 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:26:29.898196 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:26:29.898249 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:26:29.898536 (+ 287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 06:26:29.898536 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 06:26:29.898576 (+ 40us) client_negotiation.cc:769] Sending connection context
0806 06:26:29.898645 (+ 69us) client_negotiation.cc:240] Negotiation successful
0806 06:26:29.898727 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":122,"threads_started":1}
I20260806 06:26:29.898850 23955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:29.892897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.129:38635 (local address 127.21.12.190:43693)
0806 06:26:29.892939 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0806 06:26:29.892941 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:26:29.893105 (+ 164us) server_negotiation.cc:407] Connection header received
0806 06:26:29.893156 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:26:29.893157 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:26:29.893167 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:26:29.893212 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:26:29.893528 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:29.894225 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:29.895152 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:29.895335 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:29.895389 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:29.896286 (+ 897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:26:29.896287 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:26:29.896288 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 06:26:29.896294 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:26:29.897707 (+ 1413us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:26:29.898076 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:26:29.898077 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:26:29.898077 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:26:29.898136 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:26:29.898324 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:26:29.898325 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:26:29.898325 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:26:29.898457 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:26:29.898530 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0806 06:26:29.898682 (+ 152us) server_negotiation.cc:299] Negotiation successful
0806 06:26:29.898763 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
WARNING: no policy specified for kudu/127.21.12.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.12.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.12.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.130.keytab.
Entry for principal kudu/127.21.12.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.130.keytab.
I20260806 06:26:29.920684 21554 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHhLhT/build/release/bin/kudu
/tmp/dist-test-taskFHhLhT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785997581049705-21554-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785997581049705-21554-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785997581049705-21554-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785997581049705-21554-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.12.130:0
--local_ip_for_outbound_sockets=127.21.12.130
--webserver_interface=127.21.12.130
--webserver_port=0
--tserver_master_addrs=127.21.12.190:43693
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.130.keytab
--principal=kudu/127.21.12.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.148:38149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785997581049705-21554-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785997581049705-21554-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785997581049705-21554-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 06:26:30.000293 24191 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 06:26:30.000483 24191 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 06:26:30.000505 24191 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 06:26:30.001609 24191 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 06:26:30.001655 24191 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 06:26:30.001675 24191 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260806 06:26:30.001685 24191 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 06:26:30.001701 24191 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.12.130
I20260806 06:26:30.003233 24191 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.148:38149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785997581049705-21554-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785997581049705-21554-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.130.keytab
--principal=kudu/127.21.12.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.12.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785997581049705-21554-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.12.130
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.21.12.190:43693
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24191
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.12.130
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785997581049705-21554-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 05:11:02 UTC on ccycloud.quasar-hvzbbz.root.comops.site
build id 4569617
I20260806 06:26:30.003464 24191 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 06:26:30.003710 24191 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 06:26:30.008759 24202 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:26:30.008745 24199 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:26:30.008973 24191 server_base.cc:1034] running on GCE node
W20260806 06:26:30.008745 24198 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:26:30.009244 24191 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 06:26:30.009526 24191 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 06:26:30.010684 24191 hybrid_clock.cc:648] HybridClock initialized: now 1785997590010664 us; error 38 us; skew 500 ppm
Aug 06 06:26:30 dist-test-slave-4tdj krb5kdc[23821](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997590, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 06:26:30.013221 24191 init.cc:377] Logged in from keytab as kudu/127.21.12.130@KRBTEST.COM (short username kudu)
I20260806 06:26:30.013665 24191 webserver.cc:460] Webserver started at http://127.21.12.130:39201/ using document root <none> and password file <none>
I20260806 06:26:30.013892 24191 fs_manager.cc:359] Metadata directory not provided
I20260806 06:26:30.013942 24191 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 06:26:30.014053 24191 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 06:26:30.015122 24191 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785997581049705-21554-0/minicluster-data/ts-1/data/instance:
uuid: "0db93c757c334db8946a1e5df997c5d9"
format_stamp: "Formatted at 2026-08-06 06:26:30 on dist-test-slave-4tdj"
server_key: "9a736dba8d00284ddbeddeed6b5b044b"
server_key_iv: "b5d85b243fd262aa5c425c472fa5590c"
server_key_version: "encryptionkey@0"
I20260806 06:26:30.015446 24191 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785997581049705-21554-0/minicluster-data/ts-1/wal/instance:
uuid: "0db93c757c334db8946a1e5df997c5d9"
format_stamp: "Formatted at 2026-08-06 06:26:30 on dist-test-slave-4tdj"
server_key: "9a736dba8d00284ddbeddeed6b5b044b"
server_key_iv: "b5d85b243fd262aa5c425c472fa5590c"
server_key_version: "encryptionkey@0"
I20260806 06:26:30.016697 24191 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260806 06:26:30.017351 24212 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 06:26:30.017519 24191 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260806 06:26:30.017627 24191 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785997581049705-21554-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785997581049705-21554-0/minicluster-data/ts-1/wal
uuid: "0db93c757c334db8946a1e5df997c5d9"
format_stamp: "Formatted at 2026-08-06 06:26:30 on dist-test-slave-4tdj"
server_key: "9a736dba8d00284ddbeddeed6b5b044b"
server_key_iv: "b5d85b243fd262aa5c425c472fa5590c"
server_key_version: "encryptionkey@0"
I20260806 06:26:30.017679 24191 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785997581049705-21554-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785997581049705-21554-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785997581049705-21554-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 06:26:30.038630 24191 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 06:26:30.040783 24191 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 06:26:30.040897 24191 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 06:26:30.041379 24191 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 06:26:30.041422 24191 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 06:26:30.041443 24191 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 06:26:30.041455 24191 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 06:26:30.047681 24191 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.130:43279
I20260806 06:26:30.047760 24382 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.130:43279 every 8 connection(s)
I20260806 06:26:30.048087 24191 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785997581049705-21554-0/minicluster-data/ts-1/data/info.pb
Aug 06 06:26:30 dist-test-slave-4tdj krb5kdc[23821](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997590, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.130@KRBTEST.COM for kudu/127.21.12.190@KRBTEST.COM
I20260806 06:26:30.051222 23955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:30.041911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.130:50775 (local address 127.21.12.190:43693)
0806 06:26:30.041961 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0806 06:26:30.041963 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:26:30.042194 (+ 231us) server_negotiation.cc:407] Connection header received
0806 06:26:30.042762 (+ 568us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:26:30.042764 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:26:30.042778 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:26:30.042851 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:26:30.043654 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:30.044396 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:30.045811 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:30.046179 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:30.046256 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:30.048794 (+ 2538us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:26:30.048795 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:26:30.048795 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 06:26:30.048803 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:26:30.050079 (+ 1276us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:26:30.050515 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:26:30.050516 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:26:30.050516 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:26:30.050569 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:26:30.050757 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:26:30.050757 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:26:30.050757 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:26:30.050848 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:26:30.050891 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0806 06:26:30.051092 (+ 201us) server_negotiation.cc:299] Negotiation successful
0806 06:26:30.051160 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260806 06:26:30.051420 24226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:30.041915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:43693 (local address 127.21.12.130:50775)
0806 06:26:30.042077 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0806 06:26:30.042112 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0806 06:26:30.042639 (+ 527us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:26:30.042928 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:26:30.042931 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:26:30.043276 (+ 345us) client_negotiation.cc:189] Negotiated authn=SASL
0806 06:26:30.043551 (+ 275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:30.043559 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:30.044484 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:30.044485 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:30.045727 (+ 1242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:30.045730 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:30.046269 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:30.046271 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:30.046466 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:30.046980 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 06:26:30.046990 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0806 06:26:30.048690 (+ 1700us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 06:26:30.050164 (+ 1474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:26:30.050171 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:26:30.050173 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:26:30.050444 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:26:30.050616 (+ 172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:26:30.050617 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:26:30.050617 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:26:30.050716 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:26:30.050887 (+ 171us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 06:26:30.050890 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 06:26:30.050995 (+ 105us) client_negotiation.cc:769] Sending connection context
0806 06:26:30.051073 (+ 78us) client_negotiation.cc:240] Negotiation successful
0806 06:26:30.051190 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":86,"threads_started":1}
I20260806 06:26:30.051877 24384 heartbeater.cc:347] Connected to a master server at 127.21.12.190:43693
I20260806 06:26:30.051960 24384 heartbeater.cc:464] Registering TS with master...
I20260806 06:26:30.052138 24384 heartbeater.cc:511] Master 127.21.12.190:43693 requested a full tablet report, sending...
I20260806 06:26:30.052542 23883 ts_manager.cc:194] Registered new tserver with Master: 0db93c757c334db8946a1e5df997c5d9 (127.21.12.130:43279)
I20260806 06:26:30.053027 23883 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.12.130@KRBTEST.COM'} at 127.21.12.130:50775
I20260806 06:26:30.056082 21554 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHhLhT/build/release/bin/kudu as pid 24191
I20260806 06:26:30.056176 21554 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785997581049705-21554-0/minicluster-data/ts-1/wal/instance
I20260806 06:26:30.056351 21554 external_mini_cluster.cc:1428] Setting key b0594790a72a0267f1c7f4c741712e61
I20260806 06:26:30.058094 23955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:30.052884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.130:36171 (local address 127.21.12.190:43693)
0806 06:26:30.052924 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0806 06:26:30.052926 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:26:30.052969 (+ 43us) server_negotiation.cc:407] Connection header received
0806 06:26:30.053036 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:26:30.053036 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:26:30.053044 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:26:30.053088 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:26:30.053401 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:30.054003 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:30.054712 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:30.054891 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:30.054940 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:30.055732 (+ 792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:26:30.055733 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:26:30.055733 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 06:26:30.055740 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:26:30.057100 (+ 1360us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:26:30.057389 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:26:30.057390 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:26:30.057391 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:26:30.057444 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:26:30.057608 (+ 164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:26:30.057609 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:26:30.057609 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:26:30.057747 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:26:30.057804 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0806 06:26:30.057938 (+ 134us) server_negotiation.cc:299] Negotiation successful
0806 06:26:30.058008 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260806 06:26:30.058267 24391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:30.052801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:43693 (local address 127.21.12.130:36171)
0806 06:26:30.052916 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0806 06:26:30.052926 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0806 06:26:30.052989 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:26:30.053093 (+ 104us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:26:30.053094 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:26:30.053279 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0806 06:26:30.053331 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:30.053333 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:30.054086 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:30.054086 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:30.054655 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:30.054657 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:30.054972 (+ 315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:30.054973 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:30.055046 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:30.055392 (+ 346us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 06:26:30.055396 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0806 06:26:30.055657 (+ 261us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 06:26:30.057169 (+ 1512us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:26:30.057170 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:26:30.057170 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:26:30.057330 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:26:30.057495 (+ 165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:26:30.057496 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:26:30.057496 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:26:30.057554 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:26:30.057804 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 06:26:30.057805 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 06:26:30.057850 (+ 45us) client_negotiation.cc:769] Sending connection context
0806 06:26:30.057909 (+ 59us) client_negotiation.cc:240] Negotiation successful
0806 06:26:30.058009 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":89,"thread_start_us":63,"threads_started":1}
WARNING: no policy specified for kudu/127.21.12.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.12.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.12.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.131.keytab.
Entry for principal kudu/127.21.12.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.131.keytab.
I20260806 06:26:30.081446 21554 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHhLhT/build/release/bin/kudu
/tmp/dist-test-taskFHhLhT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785997581049705-21554-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785997581049705-21554-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785997581049705-21554-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785997581049705-21554-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.12.131:0
--local_ip_for_outbound_sockets=127.21.12.131
--webserver_interface=127.21.12.131
--webserver_port=0
--tserver_master_addrs=127.21.12.190:43693
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.131.keytab
--principal=kudu/127.21.12.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.148:38149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785997581049705-21554-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785997581049705-21554-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785997581049705-21554-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 06:26:30.162068 24394 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 06:26:30.162262 24394 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 06:26:30.162285 24394 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 06:26:30.163437 24394 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 06:26:30.163477 24394 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 06:26:30.163496 24394 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260806 06:26:30.163507 24394 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 06:26:30.163524 24394 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.12.131
I20260806 06:26:30.164983 24394 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.148:38149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785997581049705-21554-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785997581049705-21554-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.131.keytab
--principal=kudu/127.21.12.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.12.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785997581049705-21554-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.12.131
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.21.12.190:43693
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24394
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.12.131
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785997581049705-21554-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 05:11:02 UTC on ccycloud.quasar-hvzbbz.root.comops.site
build id 4569617
I20260806 06:26:30.165226 24394 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 06:26:30.165443 24394 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 06:26:30.170161 24409 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:26:30.170164 24411 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:26:30.170214 24408 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:26:30.170374 24394 server_base.cc:1034] running on GCE node
I20260806 06:26:30.170540 24394 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 06:26:30.170722 24394 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 06:26:30.171849 24394 hybrid_clock.cc:648] HybridClock initialized: now 1785997590171835 us; error 30 us; skew 500 ppm
Aug 06 06:26:30 dist-test-slave-4tdj krb5kdc[23821](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997590, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 06:26:30.174235 24394 init.cc:377] Logged in from keytab as kudu/127.21.12.131@KRBTEST.COM (short username kudu)
I20260806 06:26:30.174707 24394 webserver.cc:460] Webserver started at http://127.21.12.131:33759/ using document root <none> and password file <none>
I20260806 06:26:30.174923 24394 fs_manager.cc:359] Metadata directory not provided
I20260806 06:26:30.174969 24394 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 06:26:30.175093 24394 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 06:26:30.176139 24394 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785997581049705-21554-0/minicluster-data/ts-2/data/instance:
uuid: "09211067c8c14e159b2941318ccdacb5"
format_stamp: "Formatted at 2026-08-06 06:26:30 on dist-test-slave-4tdj"
server_key: "e04df719d86d2471c4656b156e2533c0"
server_key_iv: "f0e91f93b4afe06de203639a3a016ce9"
server_key_version: "encryptionkey@0"
I20260806 06:26:30.176479 24394 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785997581049705-21554-0/minicluster-data/ts-2/wal/instance:
uuid: "09211067c8c14e159b2941318ccdacb5"
format_stamp: "Formatted at 2026-08-06 06:26:30 on dist-test-slave-4tdj"
server_key: "e04df719d86d2471c4656b156e2533c0"
server_key_iv: "f0e91f93b4afe06de203639a3a016ce9"
server_key_version: "encryptionkey@0"
I20260806 06:26:30.177836 24394 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260806 06:26:30.178618 24425 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 06:26:30.178773 24394 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260806 06:26:30.178865 24394 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785997581049705-21554-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785997581049705-21554-0/minicluster-data/ts-2/wal
uuid: "09211067c8c14e159b2941318ccdacb5"
format_stamp: "Formatted at 2026-08-06 06:26:30 on dist-test-slave-4tdj"
server_key: "e04df719d86d2471c4656b156e2533c0"
server_key_iv: "f0e91f93b4afe06de203639a3a016ce9"
server_key_version: "encryptionkey@0"
I20260806 06:26:30.178925 24394 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785997581049705-21554-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785997581049705-21554-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785997581049705-21554-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 06:26:30.191941 24394 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 06:26:30.193930 24394 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 06:26:30.194064 24394 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 06:26:30.194669 24394 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 06:26:30.194720 24394 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 06:26:30.194749 24394 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 06:26:30.194765 24394 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 06:26:30 dist-test-slave-4tdj krb5kdc[23821](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997590, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.131@KRBTEST.COM for kudu/127.21.12.190@KRBTEST.COM
I20260806 06:26:30.201754 24394 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.131:34181
I20260806 06:26:30.201864 24591 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.131:34181 every 8 connection(s)
I20260806 06:26:30.202148 24394 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785997581049705-21554-0/minicluster-data/ts-2/data/info.pb
I20260806 06:26:30.203996 23955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:30.195027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.131:60341 (local address 127.21.12.190:43693)
0806 06:26:30.195080 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0806 06:26:30.195082 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:26:30.195331 (+ 249us) server_negotiation.cc:407] Connection header received
0806 06:26:30.195886 (+ 555us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:26:30.195888 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:26:30.195901 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:26:30.195981 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:26:30.196606 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:30.197220 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:30.198359 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:30.198590 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:30.198648 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:30.201439 (+ 2791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:26:30.201440 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:26:30.201441 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 06:26:30.201448 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:26:30.202899 (+ 1451us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:26:30.203262 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:26:30.203264 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:26:30.203264 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:26:30.203328 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:26:30.203528 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:26:30.203528 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:26:30.203529 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:26:30.203632 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:26:30.203676 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0806 06:26:30.203868 (+ 192us) server_negotiation.cc:299] Negotiation successful
0806 06:26:30.203931 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260806 06:26:30.204146 24439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:30.195030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:43693 (local address 127.21.12.131:60341)
0806 06:26:30.195192 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0806 06:26:30.195233 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0806 06:26:30.195790 (+ 557us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:26:30.196061 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:26:30.196065 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:26:30.196319 (+ 254us) client_negotiation.cc:189] Negotiated authn=SASL
0806 06:26:30.196516 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:30.196521 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:30.197307 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:30.197308 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:30.198268 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:30.198271 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:30.198660 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:30.198661 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:30.198776 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:30.199275 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 06:26:30.199285 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0806 06:26:30.201353 (+ 2068us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 06:26:30.202966 (+ 1613us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:26:30.202969 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:26:30.202971 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:26:30.203173 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:26:30.203388 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:26:30.203389 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:26:30.203389 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:26:30.203475 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:26:30.203672 (+ 197us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 06:26:30.203675 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 06:26:30.203781 (+ 106us) client_negotiation.cc:769] Sending connection context
0806 06:26:30.203853 (+ 72us) client_negotiation.cc:240] Negotiation successful
0806 06:26:30.203970 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":89,"threads_started":1}
I20260806 06:26:30.204586 24592 heartbeater.cc:347] Connected to a master server at 127.21.12.190:43693
I20260806 06:26:30.204664 24592 heartbeater.cc:464] Registering TS with master...
I20260806 06:26:30.204830 24592 heartbeater.cc:511] Master 127.21.12.190:43693 requested a full tablet report, sending...
I20260806 06:26:30.205153 23882 ts_manager.cc:194] Registered new tserver with Master: 09211067c8c14e159b2941318ccdacb5 (127.21.12.131:34181)
I20260806 06:26:30.205569 23882 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.12.131@KRBTEST.COM'} at 127.21.12.131:60341
I20260806 06:26:30.206161 21554 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHhLhT/build/release/bin/kudu as pid 24394
I20260806 06:26:30.206223 21554 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785997581049705-21554-0/minicluster-data/ts-2/wal/instance
I20260806 06:26:30.206405 21554 external_mini_cluster.cc:1428] Setting key ca67dd33f2470e5bee4f413f440f19ea
I20260806 06:26:30.207067 21554 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 06:26:30.210927 23955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:30.205697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.131:47221 (local address 127.21.12.190:43693)
0806 06:26:30.205763 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0806 06:26:30.205765 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:26:30.205810 (+ 45us) server_negotiation.cc:407] Connection header received
0806 06:26:30.205868 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:26:30.205868 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:26:30.205878 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:26:30.205935 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:26:30.206404 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:30.207079 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:30.207756 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:30.207950 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:30.208003 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:30.208816 (+ 813us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:26:30.208817 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:26:30.208818 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 06:26:30.208824 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:26:30.209936 (+ 1112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:26:30.210229 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:26:30.210230 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:26:30.210231 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:26:30.210282 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:26:30.210459 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:26:30.210460 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:26:30.210460 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:26:30.210574 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:26:30.210631 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0806 06:26:30.210761 (+ 130us) server_negotiation.cc:299] Negotiation successful
0806 06:26:30.210834 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260806 06:26:30.210942 24600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:30.205608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:43693 (local address 127.21.12.131:47221)
0806 06:26:30.205757 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0806 06:26:30.205767 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0806 06:26:30.205822 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:26:30.205950 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:26:30.205950 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:26:30.206174 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0806 06:26:30.206311 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:30.206313 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:30.207184 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:30.207184 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:30.207684 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:30.207686 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:30.208015 (+ 329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:30.208016 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:30.208093 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:30.208469 (+ 376us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 06:26:30.208473 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0806 06:26:30.208738 (+ 265us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 06:26:30.210013 (+ 1275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:26:30.210014 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:26:30.210014 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:26:30.210169 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:26:30.210348 (+ 179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:26:30.210349 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:26:30.210349 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:26:30.210409 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:26:30.210628 (+ 219us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 06:26:30.210629 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 06:26:30.210673 (+ 44us) client_negotiation.cc:769] Sending connection context
0806 06:26:30.210734 (+ 61us) client_negotiation.cc:240] Negotiation successful
0806 06:26:30.210836 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":62,"threads_started":1}
Aug 06 06:26:30 dist-test-slave-4tdj krb5kdc[23821](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997590, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 06 06:26:30 dist-test-slave-4tdj krb5kdc[23821](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997590, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.12.190@KRBTEST.COM
I20260806 06:26:30.229619 23955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:30.223387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47534 (local address 127.21.12.190:43693)
0806 06:26:30.223440 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0806 06:26:30.223442 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:26:30.223476 (+ 34us) server_negotiation.cc:407] Connection header received
0806 06:26:30.223501 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:26:30.223501 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:26:30.223510 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:26:30.223550 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:26:30.224094 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:30.224584 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:30.225310 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:30.225499 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:30.225554 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:30.227589 (+ 2035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:26:30.227591 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:26:30.227591 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 06:26:30.227599 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:26:30.228659 (+ 1060us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:26:30.228943 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:26:30.228944 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:26:30.228944 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:26:30.228994 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:26:30.229154 (+ 160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:26:30.229155 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:26:30.229155 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:26:30.229255 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:26:30.229341 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0806 06:26:30.229442 (+ 101us) server_negotiation.cc:299] Negotiation successful
0806 06:26:30.229528 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260806 06:26:30.230986 23882 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47534:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 06:26:30.232494 23882 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 06:26:30.238620 24615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:30.234623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.130:43279 (local address 127.0.0.1:44364)
0806 06:26:30.234904 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0806 06:26:30.234948 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0806 06:26:30.235017 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:26:30.235378 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:26:30.235379 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:26:30.235381 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 06:26:30.235436 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:30.235438 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:30.236319 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:30.236320 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:30.237465 (+ 1145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:30.237467 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:30.238330 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:30.238331 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:30.238411 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:30.238425 (+ 14us) client_negotiation.cc:769] Sending connection context
0806 06:26:30.238492 (+ 67us) client_negotiation.cc:240] Negotiation successful
0806 06:26:30.238540 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":80,"threads_started":1}
I20260806 06:26:30.238811 24616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:30.234623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.129:46063 (local address 127.0.0.1:45130)
0806 06:26:30.235045 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0806 06:26:30.235054 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0806 06:26:30.235103 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:26:30.235286 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:26:30.235289 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:26:30.235290 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 06:26:30.235357 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:30.235359 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:30.236691 (+ 1332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:30.236692 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:30.237680 (+ 988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:30.237682 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:30.238591 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:30.238593 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:30.238669 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:30.238673 (+ 4us) client_negotiation.cc:769] Sending connection context
0806 06:26:30.238703 (+ 30us) client_negotiation.cc:240] Negotiation successful
0806 06:26:30.238747 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":404,"mutex_wait_us":2,"thread_start_us":41,"threads_started":1}
I20260806 06:26:30.239068 24618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:30.234734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44364 (local address 127.21.12.130:43279)
0806 06:26:30.234987 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0806 06:26:30.234991 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 06:26:30.235001 (+ 10us) server_negotiation.cc:407] Connection header received
0806 06:26:30.235140 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:26:30.235142 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:26:30.235159 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:26:30.235244 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 06:26:30.235513 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:30.236240 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:30.237543 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:30.238213 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:30.238287 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:30.238374 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0806 06:26:30.238533 (+ 159us) server_negotiation.cc:299] Negotiation successful
0806 06:26:30.238578 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":221,"thread_start_us":79,"threads_started":1}
I20260806 06:26:30.239156 24617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:30.234717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45130 (local address 127.21.12.129:46063)
0806 06:26:30.234909 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0806 06:26:30.234914 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 06:26:30.235111 (+ 197us) server_negotiation.cc:407] Connection header received
0806 06:26:30.235157 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:26:30.235159 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:26:30.235173 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:26:30.235274 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 06:26:30.235438 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:30.236604 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:30.237770 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:30.238488 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:30.238919 (+ 431us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:30.238994 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0806 06:26:30.239019 (+ 25us) server_negotiation.cc:299] Negotiation successful
0806 06:26:30.239056 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":85,"threads_started":1}
I20260806 06:26:30.239554 24621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:30.234801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.131:34181 (local address 127.0.0.1:40218)
0806 06:26:30.235166 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0806 06:26:30.235173 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0806 06:26:30.235212 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:26:30.235439 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:26:30.235441 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:26:30.235443 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 06:26:30.235477 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:30.235478 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:30.237745 (+ 2267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:30.237745 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:30.238508 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:30.238510 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:30.239328 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:30.239329 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:30.239389 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:30.239393 (+ 4us) client_negotiation.cc:769] Sending connection context
0806 06:26:30.239450 (+ 57us) client_negotiation.cc:240] Negotiation successful
0806 06:26:30.239490 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":63,"threads_started":1}
I20260806 06:26:30.239667 24622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:30.234906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40218 (local address 127.21.12.131:34181)
0806 06:26:30.235046 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0806 06:26:30.235050 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 06:26:30.235233 (+ 183us) server_negotiation.cc:407] Connection header received
0806 06:26:30.235280 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:26:30.235283 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:26:30.235299 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:26:30.235372 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 06:26:30.235550 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:30.236618 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:30.238590 (+ 1972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:30.239181 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:30.239242 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:30.239296 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0806 06:26:30.239469 (+ 173us) server_negotiation.cc:299] Negotiation successful
0806 06:26:30.239578 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":64,"threads_started":1}
I20260806 06:26:30.239809 24291 tablet_service.cc:1467] Processing CreateTablet for tablet 9c766e7d234f4967a4cb4d67a035faa6 (DEFAULT_TABLE table=test-table [id=c7f7e74bb40e4d0bbfd4b784b10cd86c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 06:26:30.239816 24075 tablet_service.cc:1467] Processing CreateTablet for tablet 9c766e7d234f4967a4cb4d67a035faa6 (DEFAULT_TABLE table=test-table [id=c7f7e74bb40e4d0bbfd4b784b10cd86c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 06:26:30.240097 24075 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c766e7d234f4967a4cb4d67a035faa6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 06:26:30.240103 24291 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c766e7d234f4967a4cb4d67a035faa6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 06:26:30.242802 24623 tablet_bootstrap.cc:492] T 9c766e7d234f4967a4cb4d67a035faa6 P 0db93c757c334db8946a1e5df997c5d9: Bootstrap starting.
I20260806 06:26:30.243155 24501 tablet_service.cc:1467] Processing CreateTablet for tablet 9c766e7d234f4967a4cb4d67a035faa6 (DEFAULT_TABLE table=test-table [id=c7f7e74bb40e4d0bbfd4b784b10cd86c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 06:26:30.243392 24501 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c766e7d234f4967a4cb4d67a035faa6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 06:26:30.243410 24623 tablet_bootstrap.cc:654] T 9c766e7d234f4967a4cb4d67a035faa6 P 0db93c757c334db8946a1e5df997c5d9: Neither blocks nor log segments found. Creating new log.
I20260806 06:26:30.243649 24623 log.cc:824] T 9c766e7d234f4967a4cb4d67a035faa6 P 0db93c757c334db8946a1e5df997c5d9: Log is configured to *not* fsync() on all Append() calls
I20260806 06:26:30.244288 24623 tablet_bootstrap.cc:492] T 9c766e7d234f4967a4cb4d67a035faa6 P 0db93c757c334db8946a1e5df997c5d9: No bootstrap required, opened a new log
I20260806 06:26:30.244330 24623 ts_tablet_manager.cc:1397] T 9c766e7d234f4967a4cb4d67a035faa6 P 0db93c757c334db8946a1e5df997c5d9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260806 06:26:30.245697 24623 raft_consensus.cc:348] T 9c766e7d234f4967a4cb4d67a035faa6 P 0db93c757c334db8946a1e5df997c5d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0db93c757c334db8946a1e5df997c5d9" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 43279 } } peers { permanent_uuid: "09211067c8c14e159b2941318ccdacb5" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 34181 } } peers { permanent_uuid: "9447631f55b7404f9ab0232e0a1a2dc8" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 46063 } }
I20260806 06:26:30.245844 24629 tablet_bootstrap.cc:492] T 9c766e7d234f4967a4cb4d67a035faa6 P 09211067c8c14e159b2941318ccdacb5: Bootstrap starting.
I20260806 06:26:30.245870 24623 raft_consensus.cc:374] T 9c766e7d234f4967a4cb4d67a035faa6 P 0db93c757c334db8946a1e5df997c5d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 06:26:30.245908 24623 raft_consensus.cc:729] T 9c766e7d234f4967a4cb4d67a035faa6 P 0db93c757c334db8946a1e5df997c5d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0db93c757c334db8946a1e5df997c5d9, State: Initialized, Role: FOLLOWER
I20260806 06:26:30.245846 24624 tablet_bootstrap.cc:492] T 9c766e7d234f4967a4cb4d67a035faa6 P 9447631f55b7404f9ab0232e0a1a2dc8: Bootstrap starting.
I20260806 06:26:30.246038 24623 consensus_queue.cc:260] T 9c766e7d234f4967a4cb4d67a035faa6 P 0db93c757c334db8946a1e5df997c5d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0db93c757c334db8946a1e5df997c5d9" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 43279 } } peers { permanent_uuid: "09211067c8c14e159b2941318ccdacb5" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 34181 } } peers { permanent_uuid: "9447631f55b7404f9ab0232e0a1a2dc8" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 46063 } }
I20260806 06:26:30.246341 24623 ts_tablet_manager.cc:1428] T 9c766e7d234f4967a4cb4d67a035faa6 P 0db93c757c334db8946a1e5df997c5d9: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260806 06:26:30.246519 24384 heartbeater.cc:503] Master 127.21.12.190:43693 was elected leader, sending a full tablet report...
I20260806 06:26:30.246661 24624 tablet_bootstrap.cc:654] T 9c766e7d234f4967a4cb4d67a035faa6 P 9447631f55b7404f9ab0232e0a1a2dc8: Neither blocks nor log segments found. Creating new log.
I20260806 06:26:30.246732 24629 tablet_bootstrap.cc:654] T 9c766e7d234f4967a4cb4d67a035faa6 P 09211067c8c14e159b2941318ccdacb5: Neither blocks nor log segments found. Creating new log.
I20260806 06:26:30.247041 24629 log.cc:824] T 9c766e7d234f4967a4cb4d67a035faa6 P 09211067c8c14e159b2941318ccdacb5: Log is configured to *not* fsync() on all Append() calls
I20260806 06:26:30.247094 24624 log.cc:824] T 9c766e7d234f4967a4cb4d67a035faa6 P 9447631f55b7404f9ab0232e0a1a2dc8: Log is configured to *not* fsync() on all Append() calls
I20260806 06:26:30.247756 24629 tablet_bootstrap.cc:492] T 9c766e7d234f4967a4cb4d67a035faa6 P 09211067c8c14e159b2941318ccdacb5: No bootstrap required, opened a new log
I20260806 06:26:30.247774 24624 tablet_bootstrap.cc:492] T 9c766e7d234f4967a4cb4d67a035faa6 P 9447631f55b7404f9ab0232e0a1a2dc8: No bootstrap required, opened a new log
I20260806 06:26:30.247818 24624 ts_tablet_manager.cc:1397] T 9c766e7d234f4967a4cb4d67a035faa6 P 9447631f55b7404f9ab0232e0a1a2dc8: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260806 06:26:30.247818 24629 ts_tablet_manager.cc:1397] T 9c766e7d234f4967a4cb4d67a035faa6 P 09211067c8c14e159b2941318ccdacb5: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260806 06:26:30.249099 24624 raft_consensus.cc:348] T 9c766e7d234f4967a4cb4d67a035faa6 P 9447631f55b7404f9ab0232e0a1a2dc8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0db93c757c334db8946a1e5df997c5d9" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 43279 } } peers { permanent_uuid: "09211067c8c14e159b2941318ccdacb5" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 34181 } } peers { permanent_uuid: "9447631f55b7404f9ab0232e0a1a2dc8" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 46063 } }
I20260806 06:26:30.249104 24629 raft_consensus.cc:348] T 9c766e7d234f4967a4cb4d67a035faa6 P 09211067c8c14e159b2941318ccdacb5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0db93c757c334db8946a1e5df997c5d9" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 43279 } } peers { permanent_uuid: "09211067c8c14e159b2941318ccdacb5" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 34181 } } peers { permanent_uuid: "9447631f55b7404f9ab0232e0a1a2dc8" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 46063 } }
I20260806 06:26:30.249239 24629 raft_consensus.cc:374] T 9c766e7d234f4967a4cb4d67a035faa6 P 09211067c8c14e159b2941318ccdacb5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 06:26:30.249240 24624 raft_consensus.cc:374] T 9c766e7d234f4967a4cb4d67a035faa6 P 9447631f55b7404f9ab0232e0a1a2dc8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 06:26:30.249274 24629 raft_consensus.cc:729] T 9c766e7d234f4967a4cb4d67a035faa6 P 09211067c8c14e159b2941318ccdacb5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09211067c8c14e159b2941318ccdacb5, State: Initialized, Role: FOLLOWER
I20260806 06:26:30.249280 24624 raft_consensus.cc:729] T 9c766e7d234f4967a4cb4d67a035faa6 P 9447631f55b7404f9ab0232e0a1a2dc8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9447631f55b7404f9ab0232e0a1a2dc8, State: Initialized, Role: FOLLOWER
I20260806 06:26:30.249382 24624 consensus_queue.cc:260] T 9c766e7d234f4967a4cb4d67a035faa6 P 9447631f55b7404f9ab0232e0a1a2dc8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0db93c757c334db8946a1e5df997c5d9" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 43279 } } peers { permanent_uuid: "09211067c8c14e159b2941318ccdacb5" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 34181 } } peers { permanent_uuid: "9447631f55b7404f9ab0232e0a1a2dc8" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 46063 } }
I20260806 06:26:30.249382 24629 consensus_queue.cc:260] T 9c766e7d234f4967a4cb4d67a035faa6 P 09211067c8c14e159b2941318ccdacb5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0db93c757c334db8946a1e5df997c5d9" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 43279 } } peers { permanent_uuid: "09211067c8c14e159b2941318ccdacb5" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 34181 } } peers { permanent_uuid: "9447631f55b7404f9ab0232e0a1a2dc8" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 46063 } }
I20260806 06:26:30.249639 24629 ts_tablet_manager.cc:1428] T 9c766e7d234f4967a4cb4d67a035faa6 P 09211067c8c14e159b2941318ccdacb5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260806 06:26:30.249655 24624 ts_tablet_manager.cc:1428] T 9c766e7d234f4967a4cb4d67a035faa6 P 9447631f55b7404f9ab0232e0a1a2dc8: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260806 06:26:30.249686 24181 heartbeater.cc:503] Master 127.21.12.190:43693 was elected leader, sending a full tablet report...
I20260806 06:26:30.249647 24592 heartbeater.cc:503] Master 127.21.12.190:43693 was elected leader, sending a full tablet report...
I20260806 06:26:30.262764 24636 raft_consensus.cc:482] T 9c766e7d234f4967a4cb4d67a035faa6 P 9447631f55b7404f9ab0232e0a1a2dc8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 06:26:30.262882 24636 raft_consensus.cc:504] T 9c766e7d234f4967a4cb4d67a035faa6 P 9447631f55b7404f9ab0232e0a1a2dc8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0db93c757c334db8946a1e5df997c5d9" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 43279 } } peers { permanent_uuid: "09211067c8c14e159b2941318ccdacb5" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 34181 } } peers { permanent_uuid: "9447631f55b7404f9ab0232e0a1a2dc8" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 46063 } }
I20260806 06:26:30.263227 24636 leader_election.cc:288] T 9c766e7d234f4967a4cb4d67a035faa6 P 9447631f55b7404f9ab0232e0a1a2dc8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0db93c757c334db8946a1e5df997c5d9 (127.21.12.130:43279), 09211067c8c14e159b2941318ccdacb5 (127.21.12.131:34181)
I20260806 06:26:30.266152 24618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:30.263456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.129:39349 (local address 127.21.12.130:43279)
0806 06:26:30.263512 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0806 06:26:30.263513 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0806 06:26:30.263522 (+ 9us) server_negotiation.cc:407] Connection header received
0806 06:26:30.263547 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:26:30.263569 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:26:30.263578 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:26:30.263629 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 06:26:30.263821 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:30.264333 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:30.265363 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:30.265881 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:30.265956 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:30.265965 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0806 06:26:30.266055 (+ 90us) server_negotiation.cc:299] Negotiation successful
0806 06:26:30.266100 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260806 06:26:30.266172 24020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:30.263387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.130:43279 (local address 127.21.12.129:39349)
0806 06:26:30.263435 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0806 06:26:30.263446 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0806 06:26:30.263490 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:26:30.263673 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:26:30.263674 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:26:30.263676 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 06:26:30.263753 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:30.263756 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:30.264401 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:30.264402 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:30.265293 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:30.265295 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:30.265948 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:30.265949 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:30.266005 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:30.266009 (+ 4us) client_negotiation.cc:769] Sending connection context
0806 06:26:30.266063 (+ 54us) client_negotiation.cc:240] Negotiation successful
0806 06:26:30.266101 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260806 06:26:30.266284 24622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:30.263597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.129:56219 (local address 127.21.12.131:34181)
0806 06:26:30.263651 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0806 06:26:30.263653 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:26:30.263744 (+ 91us) server_negotiation.cc:407] Connection header received
0806 06:26:30.263787 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:26:30.263788 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:26:30.263798 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:26:30.263845 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 06:26:30.263969 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:30.264453 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:30.265449 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:30.265987 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:30.266047 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:30.266054 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0806 06:26:30.266189 (+ 135us) server_negotiation.cc:299] Negotiation successful
0806 06:26:30.266227 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260806 06:26:30.266469 24638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:30.263491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.131:34181 (local address 127.21.12.129:56219)
0806 06:26:30.263688 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0806 06:26:30.263697 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0806 06:26:30.263740 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:26:30.263853 (+ 113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:26:30.263854 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:26:30.263856 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 06:26:30.263917 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:30.263918 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:30.264526 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:30.264527 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:30.265377 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:30.265380 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:30.266054 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:30.266055 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:30.266123 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:30.266126 (+ 3us) client_negotiation.cc:769] Sending connection context
0806 06:26:30.266174 (+ 48us) client_negotiation.cc:240] Negotiation successful
0806 06:26:30.266396 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":124,"threads_started":1}
I20260806 06:26:30.266621 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c766e7d234f4967a4cb4d67a035faa6" candidate_uuid: "9447631f55b7404f9ab0232e0a1a2dc8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0db93c757c334db8946a1e5df997c5d9" is_pre_election: true
I20260806 06:26:30.266783 24320 raft_consensus.cc:2454] T 9c766e7d234f4967a4cb4d67a035faa6 P 0db93c757c334db8946a1e5df997c5d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9447631f55b7404f9ab0232e0a1a2dc8 in term 0.
I20260806 06:26:30.266757 24527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c766e7d234f4967a4cb4d67a035faa6" candidate_uuid: "9447631f55b7404f9ab0232e0a1a2dc8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09211067c8c14e159b2941318ccdacb5" is_pre_election: true
I20260806 06:26:30.266871 24527 raft_consensus.cc:2454] T 9c766e7d234f4967a4cb4d67a035faa6 P 09211067c8c14e159b2941318ccdacb5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9447631f55b7404f9ab0232e0a1a2dc8 in term 0.
I20260806 06:26:30.266995 24009 leader_election.cc:302] T 9c766e7d234f4967a4cb4d67a035faa6 P 9447631f55b7404f9ab0232e0a1a2dc8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0db93c757c334db8946a1e5df997c5d9, 9447631f55b7404f9ab0232e0a1a2dc8; no voters:
I20260806 06:26:30.267122 24636 raft_consensus.cc:2781] T 9c766e7d234f4967a4cb4d67a035faa6 P 9447631f55b7404f9ab0232e0a1a2dc8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 06:26:30.267181 24636 raft_consensus.cc:482] T 9c766e7d234f4967a4cb4d67a035faa6 P 9447631f55b7404f9ab0232e0a1a2dc8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 06:26:30.267204 24636 raft_consensus.cc:3037] T 9c766e7d234f4967a4cb4d67a035faa6 P 9447631f55b7404f9ab0232e0a1a2dc8 [term 0 FOLLOWER]: Advancing to term 1
I20260806 06:26:30.267877 24636 raft_consensus.cc:504] T 9c766e7d234f4967a4cb4d67a035faa6 P 9447631f55b7404f9ab0232e0a1a2dc8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0db93c757c334db8946a1e5df997c5d9" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 43279 } } peers { permanent_uuid: "09211067c8c14e159b2941318ccdacb5" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 34181 } } peers { permanent_uuid: "9447631f55b7404f9ab0232e0a1a2dc8" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 46063 } }
I20260806 06:26:30.267988 24636 leader_election.cc:288] T 9c766e7d234f4967a4cb4d67a035faa6 P 9447631f55b7404f9ab0232e0a1a2dc8 [CANDIDATE]: Term 1 election: Requested vote from peers 0db93c757c334db8946a1e5df997c5d9 (127.21.12.130:43279), 09211067c8c14e159b2941318ccdacb5 (127.21.12.131:34181)
I20260806 06:26:30.268153 24527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c766e7d234f4967a4cb4d67a035faa6" candidate_uuid: "9447631f55b7404f9ab0232e0a1a2dc8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09211067c8c14e159b2941318ccdacb5"
I20260806 06:26:30.268165 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c766e7d234f4967a4cb4d67a035faa6" candidate_uuid: "9447631f55b7404f9ab0232e0a1a2dc8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0db93c757c334db8946a1e5df997c5d9"
I20260806 06:26:30.268222 24527 raft_consensus.cc:3037] T 9c766e7d234f4967a4cb4d67a035faa6 P 09211067c8c14e159b2941318ccdacb5 [term 0 FOLLOWER]: Advancing to term 1
I20260806 06:26:30.268224 24320 raft_consensus.cc:3037] T 9c766e7d234f4967a4cb4d67a035faa6 P 0db93c757c334db8946a1e5df997c5d9 [term 0 FOLLOWER]: Advancing to term 1
I20260806 06:26:30.268832 24527 raft_consensus.cc:2454] T 9c766e7d234f4967a4cb4d67a035faa6 P 09211067c8c14e159b2941318ccdacb5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9447631f55b7404f9ab0232e0a1a2dc8 in term 1.
I20260806 06:26:30.268864 24320 raft_consensus.cc:2454] T 9c766e7d234f4967a4cb4d67a035faa6 P 0db93c757c334db8946a1e5df997c5d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9447631f55b7404f9ab0232e0a1a2dc8 in term 1.
I20260806 06:26:30.268986 24009 leader_election.cc:302] T 9c766e7d234f4967a4cb4d67a035faa6 P 9447631f55b7404f9ab0232e0a1a2dc8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09211067c8c14e159b2941318ccdacb5, 9447631f55b7404f9ab0232e0a1a2dc8; no voters:
I20260806 06:26:30.269071 24636 raft_consensus.cc:2781] T 9c766e7d234f4967a4cb4d67a035faa6 P 9447631f55b7404f9ab0232e0a1a2dc8 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 06:26:30.269207 24636 raft_consensus.cc:686] T 9c766e7d234f4967a4cb4d67a035faa6 P 9447631f55b7404f9ab0232e0a1a2dc8 [term 1 LEADER]: Becoming Leader. State: Replica: 9447631f55b7404f9ab0232e0a1a2dc8, State: Running, Role: LEADER
I20260806 06:26:30.269312 24636 consensus_queue.cc:237] T 9c766e7d234f4967a4cb4d67a035faa6 P 9447631f55b7404f9ab0232e0a1a2dc8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0db93c757c334db8946a1e5df997c5d9" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 43279 } } peers { permanent_uuid: "09211067c8c14e159b2941318ccdacb5" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 34181 } } peers { permanent_uuid: "9447631f55b7404f9ab0232e0a1a2dc8" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 46063 } }
I20260806 06:26:30.270133 23882 catalog_manager.cc:5495] T 9c766e7d234f4967a4cb4d67a035faa6 P 9447631f55b7404f9ab0232e0a1a2dc8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9447631f55b7404f9ab0232e0a1a2dc8 (127.21.12.129). New cstate: current_term: 1 leader_uuid: "9447631f55b7404f9ab0232e0a1a2dc8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0db93c757c334db8946a1e5df997c5d9" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 43279 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "09211067c8c14e159b2941318ccdacb5" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 34181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9447631f55b7404f9ab0232e0a1a2dc8" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 46063 } health_report { overall_health: HEALTHY } } }
Aug 06 06:26:30 dist-test-slave-4tdj krb5kdc[23821](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997589, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.190@KRBTEST.COM for kudu/127.21.12.190@KRBTEST.COM
I20260806 06:26:30.297117 23955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:30.290995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47540 (local address 127.21.12.190:43693)
0806 06:26:30.291093 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0806 06:26:30.291095 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:26:30.291176 (+ 81us) server_negotiation.cc:407] Connection header received
0806 06:26:30.291221 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:26:30.291221 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:26:30.291230 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:26:30.291280 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:26:30.291746 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:30.292234 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:30.292893 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:30.293080 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:30.293132 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:30.295064 (+ 1932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:26:30.295066 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:26:30.295067 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 06:26:30.295073 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:26:30.296162 (+ 1089us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:26:30.296495 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:26:30.296495 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:26:30.296495 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:26:30.296564 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:26:30.296704 (+ 140us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:26:30.296704 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:26:30.296705 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:26:30.296796 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:26:30.296840 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0806 06:26:30.296965 (+ 125us) server_negotiation.cc:299] Negotiation successful
0806 06:26:30.297030 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260806 06:26:30.297219 24657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:30.290944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:43693 (local address 127.0.0.1:47540)
0806 06:26:30.291109 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0806 06:26:30.291119 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0806 06:26:30.291181 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:26:30.291290 (+ 109us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:26:30.291291 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:26:30.291624 (+ 333us) client_negotiation.cc:189] Negotiated authn=SASL
0806 06:26:30.291672 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:30.291675 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:30.292293 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:30.292293 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:30.292832 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:30.292834 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:30.293133 (+ 299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:30.293134 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:30.293170 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:30.293611 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 06:26:30.293615 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0806 06:26:30.294987 (+ 1372us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 06:26:30.296223 (+ 1236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:26:30.296230 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:26:30.296232 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:26:30.296409 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:26:30.296611 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:26:30.296612 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:26:30.296612 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:26:30.296663 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:26:30.296835 (+ 172us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 06:26:30.296840 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 06:26:30.296879 (+ 39us) client_negotiation.cc:769] Sending connection context
0806 06:26:30.296948 (+ 69us) client_negotiation.cc:240] Negotiation successful
0806 06:26:30.297119 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"mutex_wait_us":68,"thread_start_us":98,"threads_started":1}
W20260806 06:26:30.298802 24385 tablet.cc:2369] T 9c766e7d234f4967a4cb4d67a035faa6 P 0db93c757c334db8946a1e5df997c5d9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 06:26:30.299155 23882 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.21.12.190@KRBTEST.COM'} at 127.0.0.1:47540:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260806 06:26:30.299357 23882 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 06:26:30.300813 24291 tablet_service.cc:1467] Processing CreateTablet for tablet d32d027e5a83463fa51c90774a1555de (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=63a6b2dca68d44b5933735a1efcad946]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 06:26:30.300793 24075 tablet_service.cc:1467] Processing CreateTablet for tablet d32d027e5a83463fa51c90774a1555de (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=63a6b2dca68d44b5933735a1efcad946]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 06:26:30.300814 24501 tablet_service.cc:1467] Processing CreateTablet for tablet d32d027e5a83463fa51c90774a1555de (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=63a6b2dca68d44b5933735a1efcad946]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 06:26:30.300969 24291 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d32d027e5a83463fa51c90774a1555de. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 06:26:30.300968 24075 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d32d027e5a83463fa51c90774a1555de. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 06:26:30.300969 24501 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d32d027e5a83463fa51c90774a1555de. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 06:26:30.302114 24629 tablet_bootstrap.cc:492] T d32d027e5a83463fa51c90774a1555de P 09211067c8c14e159b2941318ccdacb5: Bootstrap starting.
I20260806 06:26:30.302426 24623 tablet_bootstrap.cc:492] T d32d027e5a83463fa51c90774a1555de P 0db93c757c334db8946a1e5df997c5d9: Bootstrap starting.
I20260806 06:26:30.302682 24629 tablet_bootstrap.cc:654] T d32d027e5a83463fa51c90774a1555de P 09211067c8c14e159b2941318ccdacb5: Neither blocks nor log segments found. Creating new log.
I20260806 06:26:30.302714 24624 tablet_bootstrap.cc:492] T d32d027e5a83463fa51c90774a1555de P 9447631f55b7404f9ab0232e0a1a2dc8: Bootstrap starting.
I20260806 06:26:30.303100 24623 tablet_bootstrap.cc:654] T d32d027e5a83463fa51c90774a1555de P 0db93c757c334db8946a1e5df997c5d9: Neither blocks nor log segments found. Creating new log.
I20260806 06:26:30.303140 24624 tablet_bootstrap.cc:654] T d32d027e5a83463fa51c90774a1555de P 9447631f55b7404f9ab0232e0a1a2dc8: Neither blocks nor log segments found. Creating new log.
I20260806 06:26:30.303362 24629 tablet_bootstrap.cc:492] T d32d027e5a83463fa51c90774a1555de P 09211067c8c14e159b2941318ccdacb5: No bootstrap required, opened a new log
I20260806 06:26:30.303421 24629 ts_tablet_manager.cc:1397] T d32d027e5a83463fa51c90774a1555de P 09211067c8c14e159b2941318ccdacb5: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260806 06:26:30.303586 24629 raft_consensus.cc:348] T d32d027e5a83463fa51c90774a1555de P 09211067c8c14e159b2941318ccdacb5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9447631f55b7404f9ab0232e0a1a2dc8" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 46063 } } peers { permanent_uuid: "0db93c757c334db8946a1e5df997c5d9" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 43279 } } peers { permanent_uuid: "09211067c8c14e159b2941318ccdacb5" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 34181 } }
I20260806 06:26:30.303645 24629 raft_consensus.cc:374] T d32d027e5a83463fa51c90774a1555de P 09211067c8c14e159b2941318ccdacb5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 06:26:30.303663 24629 raft_consensus.cc:729] T d32d027e5a83463fa51c90774a1555de P 09211067c8c14e159b2941318ccdacb5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09211067c8c14e159b2941318ccdacb5, State: Initialized, Role: FOLLOWER
I20260806 06:26:30.303702 24629 consensus_queue.cc:260] T d32d027e5a83463fa51c90774a1555de P 09211067c8c14e159b2941318ccdacb5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9447631f55b7404f9ab0232e0a1a2dc8" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 46063 } } peers { permanent_uuid: "0db93c757c334db8946a1e5df997c5d9" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 43279 } } peers { permanent_uuid: "09211067c8c14e159b2941318ccdacb5" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 34181 } }
I20260806 06:26:30.303802 24629 ts_tablet_manager.cc:1428] T d32d027e5a83463fa51c90774a1555de P 09211067c8c14e159b2941318ccdacb5: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260806 06:26:30.303808 24624 tablet_bootstrap.cc:492] T d32d027e5a83463fa51c90774a1555de P 9447631f55b7404f9ab0232e0a1a2dc8: No bootstrap required, opened a new log
I20260806 06:26:30.303853 24624 ts_tablet_manager.cc:1397] T d32d027e5a83463fa51c90774a1555de P 9447631f55b7404f9ab0232e0a1a2dc8: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260806 06:26:30.303967 24623 tablet_bootstrap.cc:492] T d32d027e5a83463fa51c90774a1555de P 0db93c757c334db8946a1e5df997c5d9: No bootstrap required, opened a new log
I20260806 06:26:30.303943 24635 tablet_replica.cc:439] T d32d027e5a83463fa51c90774a1555de P 09211067c8c14e159b2941318ccdacb5: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9447631f55b7404f9ab0232e0a1a2dc8" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 46063 } } peers { permanent_uuid: "0db93c757c334db8946a1e5df997c5d9" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 43279 } } peers { permanent_uuid: "09211067c8c14e159b2941318ccdacb5" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 34181 } } }
I20260806 06:26:30.304016 24623 ts_tablet_manager.cc:1397] T d32d027e5a83463fa51c90774a1555de P 0db93c757c334db8946a1e5df997c5d9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260806 06:26:30.304023 24635 tablet_replica.cc:442] T d32d027e5a83463fa51c90774a1555de P 09211067c8c14e159b2941318ccdacb5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 06:26:30.303987 24624 raft_consensus.cc:348] T d32d027e5a83463fa51c90774a1555de P 9447631f55b7404f9ab0232e0a1a2dc8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9447631f55b7404f9ab0232e0a1a2dc8" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 46063 } } peers { permanent_uuid: "0db93c757c334db8946a1e5df997c5d9" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 43279 } } peers { permanent_uuid: "09211067c8c14e159b2941318ccdacb5" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 34181 } }
I20260806 06:26:30.304073 24624 raft_consensus.cc:374] T d32d027e5a83463fa51c90774a1555de P 9447631f55b7404f9ab0232e0a1a2dc8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 06:26:30.304093 24624 raft_consensus.cc:729] T d32d027e5a83463fa51c90774a1555de P 9447631f55b7404f9ab0232e0a1a2dc8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9447631f55b7404f9ab0232e0a1a2dc8, State: Initialized, Role: FOLLOWER
I20260806 06:26:30.304158 24624 consensus_queue.cc:260] T d32d027e5a83463fa51c90774a1555de P 9447631f55b7404f9ab0232e0a1a2dc8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9447631f55b7404f9ab0232e0a1a2dc8" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 46063 } } peers { permanent_uuid: "0db93c757c334db8946a1e5df997c5d9" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 43279 } } peers { permanent_uuid: "09211067c8c14e159b2941318ccdacb5" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 34181 } }
I20260806 06:26:30.304176 24623 raft_consensus.cc:348] T d32d027e5a83463fa51c90774a1555de P 0db93c757c334db8946a1e5df997c5d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9447631f55b7404f9ab0232e0a1a2dc8" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 46063 } } peers { permanent_uuid: "0db93c757c334db8946a1e5df997c5d9" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 43279 } } peers { permanent_uuid: "09211067c8c14e159b2941318ccdacb5" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 34181 } }
I20260806 06:26:30.304243 24623 raft_consensus.cc:374] T d32d027e5a83463fa51c90774a1555de P 0db93c757c334db8946a1e5df997c5d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 06:26:30.304266 24623 raft_consensus.cc:729] T d32d027e5a83463fa51c90774a1555de P 0db93c757c334db8946a1e5df997c5d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0db93c757c334db8946a1e5df997c5d9, State: Initialized, Role: FOLLOWER
I20260806 06:26:30.304246 24624 ts_tablet_manager.cc:1428] T d32d027e5a83463fa51c90774a1555de P 9447631f55b7404f9ab0232e0a1a2dc8: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260806 06:26:30.304306 24623 consensus_queue.cc:260] T d32d027e5a83463fa51c90774a1555de P 0db93c757c334db8946a1e5df997c5d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9447631f55b7404f9ab0232e0a1a2dc8" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 46063 } } peers { permanent_uuid: "0db93c757c334db8946a1e5df997c5d9" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 43279 } } peers { permanent_uuid: "09211067c8c14e159b2941318ccdacb5" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 34181 } }
I20260806 06:26:30.304293 24636 tablet_replica.cc:439] T d32d027e5a83463fa51c90774a1555de P 9447631f55b7404f9ab0232e0a1a2dc8: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9447631f55b7404f9ab0232e0a1a2dc8" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 46063 } } peers { permanent_uuid: "0db93c757c334db8946a1e5df997c5d9" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 43279 } } peers { permanent_uuid: "09211067c8c14e159b2941318ccdacb5" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 34181 } } }
I20260806 06:26:30.304378 24636 tablet_replica.cc:442] T d32d027e5a83463fa51c90774a1555de P 9447631f55b7404f9ab0232e0a1a2dc8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 06:26:30.304389 24623 ts_tablet_manager.cc:1428] T d32d027e5a83463fa51c90774a1555de P 0db93c757c334db8946a1e5df997c5d9: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260806 06:26:30.304601 24630 tablet_replica.cc:439] T d32d027e5a83463fa51c90774a1555de P 0db93c757c334db8946a1e5df997c5d9: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9447631f55b7404f9ab0232e0a1a2dc8" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 46063 } } peers { permanent_uuid: "0db93c757c334db8946a1e5df997c5d9" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 43279 } } peers { permanent_uuid: "09211067c8c14e159b2941318ccdacb5" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 34181 } } }
I20260806 06:26:30.304692 24630 tablet_replica.cc:442] T d32d027e5a83463fa51c90774a1555de P 0db93c757c334db8946a1e5df997c5d9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 06:26:30.349574 24636 raft_consensus.cc:482] T d32d027e5a83463fa51c90774a1555de P 9447631f55b7404f9ab0232e0a1a2dc8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 06:26:30.349674 24636 raft_consensus.cc:504] T d32d027e5a83463fa51c90774a1555de P 9447631f55b7404f9ab0232e0a1a2dc8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9447631f55b7404f9ab0232e0a1a2dc8" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 46063 } } peers { permanent_uuid: "0db93c757c334db8946a1e5df997c5d9" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 43279 } } peers { permanent_uuid: "09211067c8c14e159b2941318ccdacb5" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 34181 } }
I20260806 06:26:30.349821 24636 leader_election.cc:288] T d32d027e5a83463fa51c90774a1555de P 9447631f55b7404f9ab0232e0a1a2dc8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0db93c757c334db8946a1e5df997c5d9 (127.21.12.130:43279), 09211067c8c14e159b2941318ccdacb5 (127.21.12.131:34181)
I20260806 06:26:30.350042 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d32d027e5a83463fa51c90774a1555de" candidate_uuid: "9447631f55b7404f9ab0232e0a1a2dc8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0db93c757c334db8946a1e5df997c5d9" is_pre_election: true
I20260806 06:26:30.350050 24527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d32d027e5a83463fa51c90774a1555de" candidate_uuid: "9447631f55b7404f9ab0232e0a1a2dc8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09211067c8c14e159b2941318ccdacb5" is_pre_election: true
I20260806 06:26:30.350131 24320 raft_consensus.cc:2454] T d32d027e5a83463fa51c90774a1555de P 0db93c757c334db8946a1e5df997c5d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9447631f55b7404f9ab0232e0a1a2dc8 in term 0.
I20260806 06:26:30.350145 24527 raft_consensus.cc:2454] T d32d027e5a83463fa51c90774a1555de P 09211067c8c14e159b2941318ccdacb5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9447631f55b7404f9ab0232e0a1a2dc8 in term 0.
I20260806 06:26:30.350282 24009 leader_election.cc:302] T d32d027e5a83463fa51c90774a1555de P 9447631f55b7404f9ab0232e0a1a2dc8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09211067c8c14e159b2941318ccdacb5, 9447631f55b7404f9ab0232e0a1a2dc8; no voters:
I20260806 06:26:30.350405 24636 raft_consensus.cc:2781] T d32d027e5a83463fa51c90774a1555de P 9447631f55b7404f9ab0232e0a1a2dc8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 06:26:30.350447 24636 raft_consensus.cc:482] T d32d027e5a83463fa51c90774a1555de P 9447631f55b7404f9ab0232e0a1a2dc8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 06:26:30.350463 24636 raft_consensus.cc:3037] T d32d027e5a83463fa51c90774a1555de P 9447631f55b7404f9ab0232e0a1a2dc8 [term 0 FOLLOWER]: Advancing to term 1
I20260806 06:26:30.351111 24636 raft_consensus.cc:504] T d32d027e5a83463fa51c90774a1555de P 9447631f55b7404f9ab0232e0a1a2dc8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9447631f55b7404f9ab0232e0a1a2dc8" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 46063 } } peers { permanent_uuid: "0db93c757c334db8946a1e5df997c5d9" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 43279 } } peers { permanent_uuid: "09211067c8c14e159b2941318ccdacb5" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 34181 } }
I20260806 06:26:30.351217 24636 leader_election.cc:288] T d32d027e5a83463fa51c90774a1555de P 9447631f55b7404f9ab0232e0a1a2dc8 [CANDIDATE]: Term 1 election: Requested vote from peers 0db93c757c334db8946a1e5df997c5d9 (127.21.12.130:43279), 09211067c8c14e159b2941318ccdacb5 (127.21.12.131:34181)
I20260806 06:26:30.351410 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d32d027e5a83463fa51c90774a1555de" candidate_uuid: "9447631f55b7404f9ab0232e0a1a2dc8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0db93c757c334db8946a1e5df997c5d9"
I20260806 06:26:30.351483 24320 raft_consensus.cc:3037] T d32d027e5a83463fa51c90774a1555de P 0db93c757c334db8946a1e5df997c5d9 [term 0 FOLLOWER]: Advancing to term 1
I20260806 06:26:30.351451 24527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d32d027e5a83463fa51c90774a1555de" candidate_uuid: "9447631f55b7404f9ab0232e0a1a2dc8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09211067c8c14e159b2941318ccdacb5"
I20260806 06:26:30.351540 24527 raft_consensus.cc:3037] T d32d027e5a83463fa51c90774a1555de P 09211067c8c14e159b2941318ccdacb5 [term 0 FOLLOWER]: Advancing to term 1
I20260806 06:26:30.352170 24320 raft_consensus.cc:2454] T d32d027e5a83463fa51c90774a1555de P 0db93c757c334db8946a1e5df997c5d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9447631f55b7404f9ab0232e0a1a2dc8 in term 1.
I20260806 06:26:30.352304 24527 raft_consensus.cc:2454] T d32d027e5a83463fa51c90774a1555de P 09211067c8c14e159b2941318ccdacb5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9447631f55b7404f9ab0232e0a1a2dc8 in term 1.
I20260806 06:26:30.352326 24009 leader_election.cc:302] T d32d027e5a83463fa51c90774a1555de P 9447631f55b7404f9ab0232e0a1a2dc8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0db93c757c334db8946a1e5df997c5d9, 9447631f55b7404f9ab0232e0a1a2dc8; no voters:
I20260806 06:26:30.352454 24636 raft_consensus.cc:2781] T d32d027e5a83463fa51c90774a1555de P 9447631f55b7404f9ab0232e0a1a2dc8 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 06:26:30.352521 24636 raft_consensus.cc:686] T d32d027e5a83463fa51c90774a1555de P 9447631f55b7404f9ab0232e0a1a2dc8 [term 1 LEADER]: Becoming Leader. State: Replica: 9447631f55b7404f9ab0232e0a1a2dc8, State: Running, Role: LEADER
I20260806 06:26:30.352576 24636 consensus_queue.cc:237] T d32d027e5a83463fa51c90774a1555de P 9447631f55b7404f9ab0232e0a1a2dc8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9447631f55b7404f9ab0232e0a1a2dc8" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 46063 } } peers { permanent_uuid: "0db93c757c334db8946a1e5df997c5d9" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 43279 } } peers { permanent_uuid: "09211067c8c14e159b2941318ccdacb5" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 34181 } }
I20260806 06:26:30.352890 24639 tablet_replica.cc:439] T d32d027e5a83463fa51c90774a1555de P 9447631f55b7404f9ab0232e0a1a2dc8: TxnStatusTablet state changed. Reason: New leader 9447631f55b7404f9ab0232e0a1a2dc8. Latest consensus state: current_term: 1 leader_uuid: "9447631f55b7404f9ab0232e0a1a2dc8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9447631f55b7404f9ab0232e0a1a2dc8" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 46063 } } peers { permanent_uuid: "0db93c757c334db8946a1e5df997c5d9" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 43279 } } peers { permanent_uuid: "09211067c8c14e159b2941318ccdacb5" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 34181 } } }
I20260806 06:26:30.352996 24639 tablet_replica.cc:442] T d32d027e5a83463fa51c90774a1555de P 9447631f55b7404f9ab0232e0a1a2dc8: This TxnStatusTablet replica's current role is: LEADER
I20260806 06:26:30.353235 24664 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260806 06:26:30.353181 23881 catalog_manager.cc:5495] T d32d027e5a83463fa51c90774a1555de P 9447631f55b7404f9ab0232e0a1a2dc8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9447631f55b7404f9ab0232e0a1a2dc8 (127.21.12.129). New cstate: current_term: 1 leader_uuid: "9447631f55b7404f9ab0232e0a1a2dc8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9447631f55b7404f9ab0232e0a1a2dc8" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 46063 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0db93c757c334db8946a1e5df997c5d9" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 43279 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "09211067c8c14e159b2941318ccdacb5" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 34181 } health_report { overall_health: UNKNOWN } } }
I20260806 06:26:30.353364 24664 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260806 06:26:30.364434 24657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:30.361722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.129:46063 (local address 127.0.0.1:45136)
0806 06:26:30.361855 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0806 06:26:30.361868 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 06:26:30.361924 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:26:30.362095 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:26:30.362095 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:26:30.362096 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 06:26:30.362141 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:30.362143 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:30.363032 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:30.363032 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:30.363589 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:30.363590 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:30.364012 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:30.364012 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:30.364062 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:30.364077 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 06:26:30.364317 (+ 240us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 06:26:30.364320 (+ 3us) client_negotiation.cc:769] Sending connection context
0806 06:26:30.364343 (+ 23us) client_negotiation.cc:240] Negotiation successful
0806 06:26:30.364379 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97}
I20260806 06:26:30.364449 24617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:30.361792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45136 (local address 127.21.12.129:46063)
0806 06:26:30.361847 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0806 06:26:30.361849 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:26:30.361915 (+ 66us) server_negotiation.cc:407] Connection header received
0806 06:26:30.361995 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:26:30.361995 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:26:30.362015 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:26:30.362095 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 06:26:30.362188 (+ 93us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:30.362956 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:30.363653 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:30.363952 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:30.364016 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:30.364149 (+ 133us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 06:26:30.364266 (+ 117us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 06:26:30.364340 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0806 06:26:30.364355 (+ 15us) server_negotiation.cc:299] Negotiation successful
0806 06:26:30.364376 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260806 06:26:30.366155 24527 raft_consensus.cc:1261] T d32d027e5a83463fa51c90774a1555de P 09211067c8c14e159b2941318ccdacb5 [term 1 FOLLOWER]: Refusing update from remote peer 9447631f55b7404f9ab0232e0a1a2dc8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 06:26:30.366160 24320 raft_consensus.cc:1261] T d32d027e5a83463fa51c90774a1555de P 0db93c757c334db8946a1e5df997c5d9 [term 1 FOLLOWER]: Refusing update from remote peer 9447631f55b7404f9ab0232e0a1a2dc8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 06:26:30.366521 24636 consensus_queue.cc:1035] T d32d027e5a83463fa51c90774a1555de P 9447631f55b7404f9ab0232e0a1a2dc8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09211067c8c14e159b2941318ccdacb5" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 34181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 06:26:30.366659 24639 consensus_queue.cc:1035] T d32d027e5a83463fa51c90774a1555de P 9447631f55b7404f9ab0232e0a1a2dc8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0db93c757c334db8946a1e5df997c5d9" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 43279 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 06:26:30.367847 24635 tablet_replica.cc:439] T d32d027e5a83463fa51c90774a1555de P 09211067c8c14e159b2941318ccdacb5: TxnStatusTablet state changed. Reason: New leader 9447631f55b7404f9ab0232e0a1a2dc8. Latest consensus state: current_term: 1 leader_uuid: "9447631f55b7404f9ab0232e0a1a2dc8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9447631f55b7404f9ab0232e0a1a2dc8" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 46063 } } peers { permanent_uuid: "0db93c757c334db8946a1e5df997c5d9" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 43279 } } peers { permanent_uuid: "09211067c8c14e159b2941318ccdacb5" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 34181 } } }
I20260806 06:26:30.367950 24635 tablet_replica.cc:442] T d32d027e5a83463fa51c90774a1555de P 09211067c8c14e159b2941318ccdacb5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 06:26:30.367987 24630 tablet_replica.cc:439] T d32d027e5a83463fa51c90774a1555de P 0db93c757c334db8946a1e5df997c5d9: TxnStatusTablet state changed. Reason: New leader 9447631f55b7404f9ab0232e0a1a2dc8. Latest consensus state: current_term: 1 leader_uuid: "9447631f55b7404f9ab0232e0a1a2dc8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9447631f55b7404f9ab0232e0a1a2dc8" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 46063 } } peers { permanent_uuid: "0db93c757c334db8946a1e5df997c5d9" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 43279 } } peers { permanent_uuid: "09211067c8c14e159b2941318ccdacb5" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 34181 } } }
I20260806 06:26:30.368095 24630 tablet_replica.cc:442] T d32d027e5a83463fa51c90774a1555de P 0db93c757c334db8946a1e5df997c5d9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 06:26:30.368849 24636 tablet_replica.cc:439] T d32d027e5a83463fa51c90774a1555de P 9447631f55b7404f9ab0232e0a1a2dc8: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9447631f55b7404f9ab0232e0a1a2dc8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9447631f55b7404f9ab0232e0a1a2dc8" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 46063 } } peers { permanent_uuid: "0db93c757c334db8946a1e5df997c5d9" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 43279 } } peers { permanent_uuid: "09211067c8c14e159b2941318ccdacb5" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 34181 } } }
I20260806 06:26:30.368934 24636 tablet_replica.cc:442] T d32d027e5a83463fa51c90774a1555de P 9447631f55b7404f9ab0232e0a1a2dc8: This TxnStatusTablet replica's current role is: LEADER
I20260806 06:26:30.368988 24635 tablet_replica.cc:439] T d32d027e5a83463fa51c90774a1555de P 09211067c8c14e159b2941318ccdacb5: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9447631f55b7404f9ab0232e0a1a2dc8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9447631f55b7404f9ab0232e0a1a2dc8" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 46063 } } peers { permanent_uuid: "0db93c757c334db8946a1e5df997c5d9" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 43279 } } peers { permanent_uuid: "09211067c8c14e159b2941318ccdacb5" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 34181 } } }
I20260806 06:26:30.369037 24667 mvcc.cc:205] Tried to move back new op lower bound from 7315446130137497600 to 7315446130084519936. Current Snapshot: MvccSnapshot[applied={T|T < 7315446130137497600}]
I20260806 06:26:30.369066 24635 tablet_replica.cc:442] T d32d027e5a83463fa51c90774a1555de P 09211067c8c14e159b2941318ccdacb5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 06:26:30.369764 24639 tablet_replica.cc:439] T d32d027e5a83463fa51c90774a1555de P 9447631f55b7404f9ab0232e0a1a2dc8: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9447631f55b7404f9ab0232e0a1a2dc8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9447631f55b7404f9ab0232e0a1a2dc8" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 46063 } } peers { permanent_uuid: "0db93c757c334db8946a1e5df997c5d9" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 43279 } } peers { permanent_uuid: "09211067c8c14e159b2941318ccdacb5" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 34181 } } }
I20260806 06:26:30.369845 24639 tablet_replica.cc:442] T d32d027e5a83463fa51c90774a1555de P 9447631f55b7404f9ab0232e0a1a2dc8: This TxnStatusTablet replica's current role is: LEADER
I20260806 06:26:30.370342 24670 mvcc.cc:205] Tried to move back new op lower bound from 7315446130137497600 to 7315446130084519936. Current Snapshot: MvccSnapshot[applied={T|T < 7315446130137497600}]
I20260806 06:26:30.370421 24630 tablet_replica.cc:439] T d32d027e5a83463fa51c90774a1555de P 0db93c757c334db8946a1e5df997c5d9: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9447631f55b7404f9ab0232e0a1a2dc8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9447631f55b7404f9ab0232e0a1a2dc8" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 46063 } } peers { permanent_uuid: "0db93c757c334db8946a1e5df997c5d9" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 43279 } } peers { permanent_uuid: "09211067c8c14e159b2941318ccdacb5" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 34181 } } }
I20260806 06:26:30.370507 24630 tablet_replica.cc:442] T d32d027e5a83463fa51c90774a1555de P 0db93c757c334db8946a1e5df997c5d9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 06:26:30.371305 24669 mvcc.cc:205] Tried to move back new op lower bound from 7315446130137497600 to 7315446130084519936. Current Snapshot: MvccSnapshot[applied={T|T < 7315446130137497600}]
I20260806 06:26:30.374166 24617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:30.371213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45144 (local address 127.21.12.129:46063)
0806 06:26:30.371249 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0806 06:26:30.371250 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0806 06:26:30.371257 (+ 7us) server_negotiation.cc:407] Connection header received
0806 06:26:30.371275 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:26:30.371275 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:26:30.371286 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:26:30.371328 (+ 42us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 06:26:30.371519 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:30.372172 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:30.373320 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:30.373620 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:30.373675 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:30.373855 (+ 180us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 06:26:30.373887 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 06:26:30.373924 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0806 06:26:30.374078 (+ 154us) server_negotiation.cc:299] Negotiation successful
0806 06:26:30.374097 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260806 06:26:30.385401 24187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:30.383011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.129:46063 (local address 127.21.12.129:58091)
0806 06:26:30.383086 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0806 06:26:30.383094 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0806 06:26:30.383124 (+ 30us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:26:30.383246 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:26:30.383246 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:26:30.383247 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 06:26:30.383296 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:30.383297 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:30.384104 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:30.384104 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:30.384651 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:30.384652 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:30.385062 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:30.385062 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:30.385103 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:30.385117 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 06:26:30.385276 (+ 159us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 06:26:30.385277 (+ 1us) client_negotiation.cc:769] Sending connection context
0806 06:26:30.385306 (+ 29us) client_negotiation.cc:240] Negotiation successful
0806 06:26:30.385335 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260806 06:26:30.385407 24617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:30.383072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.129:58091 (local address 127.21.12.129:46063)
0806 06:26:30.383108 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0806 06:26:30.383110 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:26:30.383119 (+ 9us) server_negotiation.cc:407] Connection header received
0806 06:26:30.383180 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:26:30.383181 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:26:30.383190 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:26:30.383244 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 06:26:30.383331 (+ 87us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:30.384041 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:30.384713 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:30.385003 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:30.385066 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:30.385189 (+ 123us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 06:26:30.385229 (+ 40us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 06:26:30.385273 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0806 06:26:30.385311 (+ 38us) server_negotiation.cc:299] Negotiation successful
0806 06:26:30.385326 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260806 06:26:30.387728 24527 raft_consensus.cc:1261] T 9c766e7d234f4967a4cb4d67a035faa6 P 09211067c8c14e159b2941318ccdacb5 [term 1 FOLLOWER]: Refusing update from remote peer 9447631f55b7404f9ab0232e0a1a2dc8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 06:26:30.387770 24320 raft_consensus.cc:1261] T 9c766e7d234f4967a4cb4d67a035faa6 P 0db93c757c334db8946a1e5df997c5d9 [term 1 FOLLOWER]: Refusing update from remote peer 9447631f55b7404f9ab0232e0a1a2dc8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 06:26:30.387904 24639 consensus_queue.cc:1035] T 9c766e7d234f4967a4cb4d67a035faa6 P 9447631f55b7404f9ab0232e0a1a2dc8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09211067c8c14e159b2941318ccdacb5" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 34181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 06:26:30.388000 24636 consensus_queue.cc:1035] T 9c766e7d234f4967a4cb4d67a035faa6 P 9447631f55b7404f9ab0232e0a1a2dc8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0db93c757c334db8946a1e5df997c5d9" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 43279 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 06:26:30.399423 23881 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47534:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 06:26:30.399557 23881 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47534:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 06:26:30.400272 23881 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b3579394671149a89ff9b5c9f4b9854e: Sending DeleteTablet for 3 replicas of tablet 9c766e7d234f4967a4cb4d67a035faa6
I20260806 06:26:30.400600 24074 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c766e7d234f4967a4cb4d67a035faa6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 06:26:30 UTC) from {username='kudu', principal='kudu/127.21.12.190@KRBTEST.COM'} at 127.0.0.1:45130
I20260806 06:26:30.400628 24501 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c766e7d234f4967a4cb4d67a035faa6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 06:26:30 UTC) from {username='kudu', principal='kudu/127.21.12.190@KRBTEST.COM'} at 127.0.0.1:40218
I20260806 06:26:30.400718 24291 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c766e7d234f4967a4cb4d67a035faa6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 06:26:30 UTC) from {username='kudu', principal='kudu/127.21.12.190@KRBTEST.COM'} at 127.0.0.1:44364
I20260806 06:26:30.400877 24689 tablet_replica.cc:330] T 9c766e7d234f4967a4cb4d67a035faa6 P 9447631f55b7404f9ab0232e0a1a2dc8: stopping tablet replica
I20260806 06:26:30.400985 24689 raft_consensus.cc:2229] T 9c766e7d234f4967a4cb4d67a035faa6 P 9447631f55b7404f9ab0232e0a1a2dc8 [term 1 LEADER]: Raft consensus shutting down.
I20260806 06:26:30.401041 24690 tablet_replica.cc:330] T 9c766e7d234f4967a4cb4d67a035faa6 P 09211067c8c14e159b2941318ccdacb5: stopping tablet replica
I20260806 06:26:30.401018 23881 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47534:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260806 06:26:30.401093 24689 raft_consensus.cc:2258] T 9c766e7d234f4967a4cb4d67a035faa6 P 9447631f55b7404f9ab0232e0a1a2dc8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 06:26:30.401144 24690 raft_consensus.cc:2229] T 9c766e7d234f4967a4cb4d67a035faa6 P 09211067c8c14e159b2941318ccdacb5 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260806 06:26:30.401151 23881 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 06:26:30.401194 24690 raft_consensus.cc:2258] T 9c766e7d234f4967a4cb4d67a035faa6 P 09211067c8c14e159b2941318ccdacb5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 06:26:30.401667 24690 ts_tablet_manager.cc:1905] T 9c766e7d234f4967a4cb4d67a035faa6 P 09211067c8c14e159b2941318ccdacb5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 06:26:30.402361 24689 ts_tablet_manager.cc:1905] T 9c766e7d234f4967a4cb4d67a035faa6 P 9447631f55b7404f9ab0232e0a1a2dc8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 06:26:30.402428 24074 tablet_service.cc:1467] Processing CreateTablet for tablet 5a8efde1df11492d9d5c93cfc388816e (DEFAULT_TABLE table=test-table [id=43237d177a73411588243d764e65869c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 06:26:30.402540 24074 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a8efde1df11492d9d5c93cfc388816e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 06:26:30.403442 24690 ts_tablet_manager.cc:1918] T 9c766e7d234f4967a4cb4d67a035faa6 P 09211067c8c14e159b2941318ccdacb5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 06:26:30.403518 24690 log.cc:1196] T 9c766e7d234f4967a4cb4d67a035faa6 P 09211067c8c14e159b2941318ccdacb5: Deleting WAL directory at /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785997581049705-21554-0/minicluster-data/ts-2/wal/wals/9c766e7d234f4967a4cb4d67a035faa6
I20260806 06:26:30.403759 24690 ts_tablet_manager.cc:1939] T 9c766e7d234f4967a4cb4d67a035faa6 P 09211067c8c14e159b2941318ccdacb5: Deleting consensus metadata
I20260806 06:26:30.403894 24624 tablet_bootstrap.cc:492] T 5a8efde1df11492d9d5c93cfc388816e P 9447631f55b7404f9ab0232e0a1a2dc8: Bootstrap starting.
I20260806 06:26:30.404065 24689 ts_tablet_manager.cc:1918] T 9c766e7d234f4967a4cb4d67a035faa6 P 9447631f55b7404f9ab0232e0a1a2dc8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 06:26:30.404127 24689 log.cc:1196] T 9c766e7d234f4967a4cb4d67a035faa6 P 9447631f55b7404f9ab0232e0a1a2dc8: Deleting WAL directory at /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785997581049705-21554-0/minicluster-data/ts-0/wal/wals/9c766e7d234f4967a4cb4d67a035faa6
I20260806 06:26:30.404260 24624 tablet_bootstrap.cc:654] T 5a8efde1df11492d9d5c93cfc388816e P 9447631f55b7404f9ab0232e0a1a2dc8: Neither blocks nor log segments found. Creating new log.
I20260806 06:26:30.404356 24689 ts_tablet_manager.cc:1939] T 9c766e7d234f4967a4cb4d67a035faa6 P 9447631f55b7404f9ab0232e0a1a2dc8: Deleting consensus metadata
I20260806 06:26:30.404785 23862 catalog_manager.cc:4841] TS 09211067c8c14e159b2941318ccdacb5 (127.21.12.131:34181): tablet 9c766e7d234f4967a4cb4d67a035faa6 (table test-table [id=c7f7e74bb40e4d0bbfd4b784b10cd86c]) successfully deleted
I20260806 06:26:30.404870 24624 tablet_bootstrap.cc:492] T 5a8efde1df11492d9d5c93cfc388816e P 9447631f55b7404f9ab0232e0a1a2dc8: No bootstrap required, opened a new log
I20260806 06:26:30.404904 24624 ts_tablet_manager.cc:1397] T 5a8efde1df11492d9d5c93cfc388816e P 9447631f55b7404f9ab0232e0a1a2dc8: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260806 06:26:30.404908 23865 catalog_manager.cc:4841] TS 9447631f55b7404f9ab0232e0a1a2dc8 (127.21.12.129:46063): tablet 9c766e7d234f4967a4cb4d67a035faa6 (table test-table [id=c7f7e74bb40e4d0bbfd4b784b10cd86c]) successfully deleted
I20260806 06:26:30.405032 24624 raft_consensus.cc:348] T 5a8efde1df11492d9d5c93cfc388816e P 9447631f55b7404f9ab0232e0a1a2dc8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9447631f55b7404f9ab0232e0a1a2dc8" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 46063 } } peers { permanent_uuid: "0db93c757c334db8946a1e5df997c5d9" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 43279 } } peers { permanent_uuid: "09211067c8c14e159b2941318ccdacb5" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 34181 } }
I20260806 06:26:30.405112 24624 raft_consensus.cc:374] T 5a8efde1df11492d9d5c93cfc388816e P 9447631f55b7404f9ab0232e0a1a2dc8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 06:26:30.405129 24624 raft_consensus.cc:729] T 5a8efde1df11492d9d5c93cfc388816e P 9447631f55b7404f9ab0232e0a1a2dc8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9447631f55b7404f9ab0232e0a1a2dc8, State: Initialized, Role: FOLLOWER
I20260806 06:26:30.405151 24501 tablet_service.cc:1467] Processing CreateTablet for tablet 5a8efde1df11492d9d5c93cfc388816e (DEFAULT_TABLE table=test-table [id=43237d177a73411588243d764e65869c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 06:26:30.405171 24624 consensus_queue.cc:260] T 5a8efde1df11492d9d5c93cfc388816e P 9447631f55b7404f9ab0232e0a1a2dc8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9447631f55b7404f9ab0232e0a1a2dc8" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 46063 } } peers { permanent_uuid: "0db93c757c334db8946a1e5df997c5d9" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 43279 } } peers { permanent_uuid: "09211067c8c14e159b2941318ccdacb5" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 34181 } }
I20260806 06:26:30.405260 24501 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a8efde1df11492d9d5c93cfc388816e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 06:26:30.405272 24624 ts_tablet_manager.cc:1428] T 5a8efde1df11492d9d5c93cfc388816e P 9447631f55b7404f9ab0232e0a1a2dc8: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260806 06:26:30.406167 24629 tablet_bootstrap.cc:492] T 5a8efde1df11492d9d5c93cfc388816e P 09211067c8c14e159b2941318ccdacb5: Bootstrap starting.
I20260806 06:26:30.406579 24629 tablet_bootstrap.cc:654] T 5a8efde1df11492d9d5c93cfc388816e P 09211067c8c14e159b2941318ccdacb5: Neither blocks nor log segments found. Creating new log.
I20260806 06:26:30.406939 24291 tablet_service.cc:1467] Processing CreateTablet for tablet 5a8efde1df11492d9d5c93cfc388816e (DEFAULT_TABLE table=test-table [id=43237d177a73411588243d764e65869c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 06:26:30.407061 24291 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a8efde1df11492d9d5c93cfc388816e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 06:26:30.407930 24623 tablet_bootstrap.cc:492] T 5a8efde1df11492d9d5c93cfc388816e P 0db93c757c334db8946a1e5df997c5d9: Bootstrap starting.
I20260806 06:26:30.408327 24623 tablet_bootstrap.cc:654] T 5a8efde1df11492d9d5c93cfc388816e P 0db93c757c334db8946a1e5df997c5d9: Neither blocks nor log segments found. Creating new log.
I20260806 06:26:30.408623 24629 tablet_bootstrap.cc:492] T 5a8efde1df11492d9d5c93cfc388816e P 09211067c8c14e159b2941318ccdacb5: No bootstrap required, opened a new log
I20260806 06:26:30.408672 24629 ts_tablet_manager.cc:1397] T 5a8efde1df11492d9d5c93cfc388816e P 09211067c8c14e159b2941318ccdacb5: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.000s
I20260806 06:26:30.408813 24629 raft_consensus.cc:348] T 5a8efde1df11492d9d5c93cfc388816e P 09211067c8c14e159b2941318ccdacb5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9447631f55b7404f9ab0232e0a1a2dc8" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 46063 } } peers { permanent_uuid: "0db93c757c334db8946a1e5df997c5d9" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 43279 } } peers { permanent_uuid: "09211067c8c14e159b2941318ccdacb5" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 34181 } }
I20260806 06:26:30.408883 24629 raft_consensus.cc:374] T 5a8efde1df11492d9d5c93cfc388816e P 09211067c8c14e159b2941318ccdacb5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 06:26:30.408903 24629 raft_consensus.cc:729] T 5a8efde1df11492d9d5c93cfc388816e P 09211067c8c14e159b2941318ccdacb5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09211067c8c14e159b2941318ccdacb5, State: Initialized, Role: FOLLOWER
I20260806 06:26:30.408921 24623 tablet_bootstrap.cc:492] T 5a8efde1df11492d9d5c93cfc388816e P 0db93c757c334db8946a1e5df997c5d9: No bootstrap required, opened a new log
I20260806 06:26:30.408941 24629 consensus_queue.cc:260] T 5a8efde1df11492d9d5c93cfc388816e P 09211067c8c14e159b2941318ccdacb5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9447631f55b7404f9ab0232e0a1a2dc8" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 46063 } } peers { permanent_uuid: "0db93c757c334db8946a1e5df997c5d9" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 43279 } } peers { permanent_uuid: "09211067c8c14e159b2941318ccdacb5" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 34181 } }
I20260806 06:26:30.408967 24623 ts_tablet_manager.cc:1397] T 5a8efde1df11492d9d5c93cfc388816e P 0db93c757c334db8946a1e5df997c5d9: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260806 06:26:30.409030 24629 ts_tablet_manager.cc:1428] T 5a8efde1df11492d9d5c93cfc388816e P 09211067c8c14e159b2941318ccdacb5: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260806 06:26:30.409104 24623 raft_consensus.cc:348] T 5a8efde1df11492d9d5c93cfc388816e P 0db93c757c334db8946a1e5df997c5d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9447631f55b7404f9ab0232e0a1a2dc8" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 46063 } } peers { permanent_uuid: "0db93c757c334db8946a1e5df997c5d9" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 43279 } } peers { permanent_uuid: "09211067c8c14e159b2941318ccdacb5" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 34181 } }
I20260806 06:26:30.409176 24623 raft_consensus.cc:374] T 5a8efde1df11492d9d5c93cfc388816e P 0db93c757c334db8946a1e5df997c5d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 06:26:30.409199 24623 raft_consensus.cc:729] T 5a8efde1df11492d9d5c93cfc388816e P 0db93c757c334db8946a1e5df997c5d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0db93c757c334db8946a1e5df997c5d9, State: Initialized, Role: FOLLOWER
I20260806 06:26:30.409240 24623 consensus_queue.cc:260] T 5a8efde1df11492d9d5c93cfc388816e P 0db93c757c334db8946a1e5df997c5d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9447631f55b7404f9ab0232e0a1a2dc8" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 46063 } } peers { permanent_uuid: "0db93c757c334db8946a1e5df997c5d9" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 43279 } } peers { permanent_uuid: "09211067c8c14e159b2941318ccdacb5" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 34181 } }
I20260806 06:26:30.409324 24623 ts_tablet_manager.cc:1428] T 5a8efde1df11492d9d5c93cfc388816e P 0db93c757c334db8946a1e5df997c5d9: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260806 06:26:30.410382 24691 tablet_replica.cc:330] T 9c766e7d234f4967a4cb4d67a035faa6 P 0db93c757c334db8946a1e5df997c5d9: stopping tablet replica
I20260806 06:26:30.410491 24691 raft_consensus.cc:2229] T 9c766e7d234f4967a4cb4d67a035faa6 P 0db93c757c334db8946a1e5df997c5d9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 06:26:30.410555 24691 raft_consensus.cc:2258] T 9c766e7d234f4967a4cb4d67a035faa6 P 0db93c757c334db8946a1e5df997c5d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 06:26:30.410926 24691 ts_tablet_manager.cc:1905] T 9c766e7d234f4967a4cb4d67a035faa6 P 0db93c757c334db8946a1e5df997c5d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 06:26:30.412169 24691 ts_tablet_manager.cc:1918] T 9c766e7d234f4967a4cb4d67a035faa6 P 0db93c757c334db8946a1e5df997c5d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 06:26:30.412228 24691 log.cc:1196] T 9c766e7d234f4967a4cb4d67a035faa6 P 0db93c757c334db8946a1e5df997c5d9: Deleting WAL directory at /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785997581049705-21554-0/minicluster-data/ts-1/wal/wals/9c766e7d234f4967a4cb4d67a035faa6
I20260806 06:26:30.412417 24691 ts_tablet_manager.cc:1939] T 9c766e7d234f4967a4cb4d67a035faa6 P 0db93c757c334db8946a1e5df997c5d9: Deleting consensus metadata
I20260806 06:26:30.412760 23862 catalog_manager.cc:4841] TS 0db93c757c334db8946a1e5df997c5d9 (127.21.12.130:43279): tablet 9c766e7d234f4967a4cb4d67a035faa6 (table test-table [id=c7f7e74bb40e4d0bbfd4b784b10cd86c]) successfully deleted
I20260806 06:26:30.434388 24636 raft_consensus.cc:482] T 5a8efde1df11492d9d5c93cfc388816e P 9447631f55b7404f9ab0232e0a1a2dc8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 06:26:30.434496 24636 raft_consensus.cc:504] T 5a8efde1df11492d9d5c93cfc388816e P 9447631f55b7404f9ab0232e0a1a2dc8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9447631f55b7404f9ab0232e0a1a2dc8" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 46063 } } peers { permanent_uuid: "0db93c757c334db8946a1e5df997c5d9" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 43279 } } peers { permanent_uuid: "09211067c8c14e159b2941318ccdacb5" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 34181 } }
I20260806 06:26:30.434644 24636 leader_election.cc:288] T 5a8efde1df11492d9d5c93cfc388816e P 9447631f55b7404f9ab0232e0a1a2dc8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0db93c757c334db8946a1e5df997c5d9 (127.21.12.130:43279), 09211067c8c14e159b2941318ccdacb5 (127.21.12.131:34181)
I20260806 06:26:30.434896 24527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a8efde1df11492d9d5c93cfc388816e" candidate_uuid: "9447631f55b7404f9ab0232e0a1a2dc8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09211067c8c14e159b2941318ccdacb5" is_pre_election: true
I20260806 06:26:30.435003 24527 raft_consensus.cc:2454] T 5a8efde1df11492d9d5c93cfc388816e P 09211067c8c14e159b2941318ccdacb5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9447631f55b7404f9ab0232e0a1a2dc8 in term 0.
I20260806 06:26:30.435166 24009 leader_election.cc:302] T 5a8efde1df11492d9d5c93cfc388816e P 9447631f55b7404f9ab0232e0a1a2dc8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09211067c8c14e159b2941318ccdacb5, 9447631f55b7404f9ab0232e0a1a2dc8; no voters:
I20260806 06:26:30.435261 24636 raft_consensus.cc:2781] T 5a8efde1df11492d9d5c93cfc388816e P 9447631f55b7404f9ab0232e0a1a2dc8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 06:26:30.435297 24636 raft_consensus.cc:482] T 5a8efde1df11492d9d5c93cfc388816e P 9447631f55b7404f9ab0232e0a1a2dc8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 06:26:30.435320 24636 raft_consensus.cc:3037] T 5a8efde1df11492d9d5c93cfc388816e P 9447631f55b7404f9ab0232e0a1a2dc8 [term 0 FOLLOWER]: Advancing to term 1
I20260806 06:26:30.436177 24636 raft_consensus.cc:504] T 5a8efde1df11492d9d5c93cfc388816e P 9447631f55b7404f9ab0232e0a1a2dc8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9447631f55b7404f9ab0232e0a1a2dc8" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 46063 } } peers { permanent_uuid: "0db93c757c334db8946a1e5df997c5d9" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 43279 } } peers { permanent_uuid: "09211067c8c14e159b2941318ccdacb5" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 34181 } }
I20260806 06:26:30.436304 24636 leader_election.cc:288] T 5a8efde1df11492d9d5c93cfc388816e P 9447631f55b7404f9ab0232e0a1a2dc8 [CANDIDATE]: Term 1 election: Requested vote from peers 0db93c757c334db8946a1e5df997c5d9 (127.21.12.130:43279), 09211067c8c14e159b2941318ccdacb5 (127.21.12.131:34181)
I20260806 06:26:30.436549 24527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a8efde1df11492d9d5c93cfc388816e" candidate_uuid: "9447631f55b7404f9ab0232e0a1a2dc8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09211067c8c14e159b2941318ccdacb5"
I20260806 06:26:30.436638 24527 raft_consensus.cc:3037] T 5a8efde1df11492d9d5c93cfc388816e P 09211067c8c14e159b2941318ccdacb5 [term 0 FOLLOWER]: Advancing to term 1
I20260806 06:26:30.436657 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a8efde1df11492d9d5c93cfc388816e" candidate_uuid: "9447631f55b7404f9ab0232e0a1a2dc8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0db93c757c334db8946a1e5df997c5d9" is_pre_election: true
I20260806 06:26:30.436748 24320 raft_consensus.cc:2454] T 5a8efde1df11492d9d5c93cfc388816e P 0db93c757c334db8946a1e5df997c5d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9447631f55b7404f9ab0232e0a1a2dc8 in term 0.
I20260806 06:26:30.436839 24319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a8efde1df11492d9d5c93cfc388816e" candidate_uuid: "9447631f55b7404f9ab0232e0a1a2dc8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0db93c757c334db8946a1e5df997c5d9"
I20260806 06:26:30.436902 24319 raft_consensus.cc:3037] T 5a8efde1df11492d9d5c93cfc388816e P 0db93c757c334db8946a1e5df997c5d9 [term 0 FOLLOWER]: Advancing to term 1
I20260806 06:26:30.437433 24527 raft_consensus.cc:2454] T 5a8efde1df11492d9d5c93cfc388816e P 09211067c8c14e159b2941318ccdacb5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9447631f55b7404f9ab0232e0a1a2dc8 in term 1.
I20260806 06:26:30.437614 24009 leader_election.cc:302] T 5a8efde1df11492d9d5c93cfc388816e P 9447631f55b7404f9ab0232e0a1a2dc8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09211067c8c14e159b2941318ccdacb5, 9447631f55b7404f9ab0232e0a1a2dc8; no voters:
I20260806 06:26:30.438102 24319 raft_consensus.cc:2454] T 5a8efde1df11492d9d5c93cfc388816e P 0db93c757c334db8946a1e5df997c5d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9447631f55b7404f9ab0232e0a1a2dc8 in term 1.
I20260806 06:26:30.438277 24636 raft_consensus.cc:2781] T 5a8efde1df11492d9d5c93cfc388816e P 9447631f55b7404f9ab0232e0a1a2dc8 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 06:26:30.438350 24636 raft_consensus.cc:686] T 5a8efde1df11492d9d5c93cfc388816e P 9447631f55b7404f9ab0232e0a1a2dc8 [term 1 LEADER]: Becoming Leader. State: Replica: 9447631f55b7404f9ab0232e0a1a2dc8, State: Running, Role: LEADER
I20260806 06:26:30.438403 24636 consensus_queue.cc:237] T 5a8efde1df11492d9d5c93cfc388816e P 9447631f55b7404f9ab0232e0a1a2dc8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9447631f55b7404f9ab0232e0a1a2dc8" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 46063 } } peers { permanent_uuid: "0db93c757c334db8946a1e5df997c5d9" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 43279 } } peers { permanent_uuid: "09211067c8c14e159b2941318ccdacb5" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 34181 } }
I20260806 06:26:30.438978 23880 catalog_manager.cc:5495] T 5a8efde1df11492d9d5c93cfc388816e P 9447631f55b7404f9ab0232e0a1a2dc8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9447631f55b7404f9ab0232e0a1a2dc8 (127.21.12.129). New cstate: current_term: 1 leader_uuid: "9447631f55b7404f9ab0232e0a1a2dc8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9447631f55b7404f9ab0232e0a1a2dc8" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 46063 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0db93c757c334db8946a1e5df997c5d9" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 43279 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "09211067c8c14e159b2941318ccdacb5" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 34181 } health_report { overall_health: UNKNOWN } } }
I20260806 06:26:30.447608 24527 raft_consensus.cc:1261] T 5a8efde1df11492d9d5c93cfc388816e P 09211067c8c14e159b2941318ccdacb5 [term 1 FOLLOWER]: Refusing update from remote peer 9447631f55b7404f9ab0232e0a1a2dc8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 06:26:30.447680 24319 raft_consensus.cc:1261] T 5a8efde1df11492d9d5c93cfc388816e P 0db93c757c334db8946a1e5df997c5d9 [term 1 FOLLOWER]: Refusing update from remote peer 9447631f55b7404f9ab0232e0a1a2dc8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 06:26:30.447778 24678 consensus_queue.cc:1035] T 5a8efde1df11492d9d5c93cfc388816e P 9447631f55b7404f9ab0232e0a1a2dc8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09211067c8c14e159b2941318ccdacb5" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 34181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 06:26:30.447876 24678 consensus_queue.cc:1035] T 5a8efde1df11492d9d5c93cfc388816e P 9447631f55b7404f9ab0232e0a1a2dc8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0db93c757c334db8946a1e5df997c5d9" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 43279 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260806 06:26:30.452720 24667 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260806 06:26:31.969586 23880 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47534:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 06:26:31.969682 23880 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47534:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 06:26:31.970484 23880 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b3579394671149a89ff9b5c9f4b9854e: Sending DeleteTablet for 3 replicas of tablet 5a8efde1df11492d9d5c93cfc388816e
I20260806 06:26:31.970713 24074 tablet_service.cc:1514] Processing DeleteTablet for tablet 5a8efde1df11492d9d5c93cfc388816e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 06:26:31 UTC) from {username='kudu', principal='kudu/127.21.12.190@KRBTEST.COM'} at 127.0.0.1:45130
I20260806 06:26:31.970726 24291 tablet_service.cc:1514] Processing DeleteTablet for tablet 5a8efde1df11492d9d5c93cfc388816e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 06:26:31 UTC) from {username='kudu', principal='kudu/127.21.12.190@KRBTEST.COM'} at 127.0.0.1:44364
I20260806 06:26:31.970916 24732 tablet_replica.cc:330] T 5a8efde1df11492d9d5c93cfc388816e P 9447631f55b7404f9ab0232e0a1a2dc8: stopping tablet replica
I20260806 06:26:31.970981 24732 raft_consensus.cc:2229] T 5a8efde1df11492d9d5c93cfc388816e P 9447631f55b7404f9ab0232e0a1a2dc8 [term 1 LEADER]: Raft consensus shutting down.
I20260806 06:26:31.970986 24501 tablet_service.cc:1514] Processing DeleteTablet for tablet 5a8efde1df11492d9d5c93cfc388816e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 06:26:31 UTC) from {username='kudu', principal='kudu/127.21.12.190@KRBTEST.COM'} at 127.0.0.1:40218
I20260806 06:26:31.971041 24732 raft_consensus.cc:2258] T 5a8efde1df11492d9d5c93cfc388816e P 9447631f55b7404f9ab0232e0a1a2dc8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 06:26:31.971166 24734 tablet_replica.cc:330] T 5a8efde1df11492d9d5c93cfc388816e P 09211067c8c14e159b2941318ccdacb5: stopping tablet replica
I20260806 06:26:31.971227 24734 raft_consensus.cc:2229] T 5a8efde1df11492d9d5c93cfc388816e P 09211067c8c14e159b2941318ccdacb5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 06:26:31.971266 24734 raft_consensus.cc:2258] T 5a8efde1df11492d9d5c93cfc388816e P 09211067c8c14e159b2941318ccdacb5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 06:26:31.971302 24732 ts_tablet_manager.cc:1905] T 5a8efde1df11492d9d5c93cfc388816e P 9447631f55b7404f9ab0232e0a1a2dc8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 06:26:31.971449 24734 ts_tablet_manager.cc:1905] T 5a8efde1df11492d9d5c93cfc388816e P 09211067c8c14e159b2941318ccdacb5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 06:26:31.973068 24730 tablet_replica.cc:330] T 5a8efde1df11492d9d5c93cfc388816e P 0db93c757c334db8946a1e5df997c5d9: stopping tablet replica
I20260806 06:26:31.973124 24730 raft_consensus.cc:2229] T 5a8efde1df11492d9d5c93cfc388816e P 0db93c757c334db8946a1e5df997c5d9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 06:26:31.973202 24730 raft_consensus.cc:2258] T 5a8efde1df11492d9d5c93cfc388816e P 0db93c757c334db8946a1e5df997c5d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 06:26:31.973323 24732 ts_tablet_manager.cc:1918] T 5a8efde1df11492d9d5c93cfc388816e P 9447631f55b7404f9ab0232e0a1a2dc8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260806 06:26:31.973371 24732 log.cc:1196] T 5a8efde1df11492d9d5c93cfc388816e P 9447631f55b7404f9ab0232e0a1a2dc8: Deleting WAL directory at /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785997581049705-21554-0/minicluster-data/ts-0/wal/wals/5a8efde1df11492d9d5c93cfc388816e
I20260806 06:26:31.973658 24732 ts_tablet_manager.cc:1939] T 5a8efde1df11492d9d5c93cfc388816e P 9447631f55b7404f9ab0232e0a1a2dc8: Deleting consensus metadata
I20260806 06:26:31.973918 24734 ts_tablet_manager.cc:1918] T 5a8efde1df11492d9d5c93cfc388816e P 09211067c8c14e159b2941318ccdacb5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260806 06:26:31.973935 23865 catalog_manager.cc:4841] TS 9447631f55b7404f9ab0232e0a1a2dc8 (127.21.12.129:46063): tablet 5a8efde1df11492d9d5c93cfc388816e (table test-table [id=43237d177a73411588243d764e65869c]) successfully deleted
I20260806 06:26:31.973977 24734 log.cc:1196] T 5a8efde1df11492d9d5c93cfc388816e P 09211067c8c14e159b2941318ccdacb5: Deleting WAL directory at /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785997581049705-21554-0/minicluster-data/ts-2/wal/wals/5a8efde1df11492d9d5c93cfc388816e
I20260806 06:26:31.974154 24734 ts_tablet_manager.cc:1939] T 5a8efde1df11492d9d5c93cfc388816e P 09211067c8c14e159b2941318ccdacb5: Deleting consensus metadata
I20260806 06:26:31.974432 23862 catalog_manager.cc:4841] TS 09211067c8c14e159b2941318ccdacb5 (127.21.12.131:34181): tablet 5a8efde1df11492d9d5c93cfc388816e (table test-table [id=43237d177a73411588243d764e65869c]) successfully deleted
I20260806 06:26:31.974529 24730 ts_tablet_manager.cc:1905] T 5a8efde1df11492d9d5c93cfc388816e P 0db93c757c334db8946a1e5df997c5d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 06:26:31.975720 24730 ts_tablet_manager.cc:1918] T 5a8efde1df11492d9d5c93cfc388816e P 0db93c757c334db8946a1e5df997c5d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260806 06:26:31.975768 24730 log.cc:1196] T 5a8efde1df11492d9d5c93cfc388816e P 0db93c757c334db8946a1e5df997c5d9: Deleting WAL directory at /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785997581049705-21554-0/minicluster-data/ts-1/wal/wals/5a8efde1df11492d9d5c93cfc388816e
I20260806 06:26:31.975912 24730 ts_tablet_manager.cc:1939] T 5a8efde1df11492d9d5c93cfc388816e P 0db93c757c334db8946a1e5df997c5d9: Deleting consensus metadata
I20260806 06:26:31.976277 23862 catalog_manager.cc:4841] TS 0db93c757c334db8946a1e5df997c5d9 (127.21.12.130:43279): tablet 5a8efde1df11492d9d5c93cfc388816e (table test-table [id=43237d177a73411588243d764e65869c]) successfully deleted
Aug 06 06:26:31 dist-test-slave-4tdj krb5kdc[23821](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997591, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 06 06:26:31 dist-test-slave-4tdj krb5kdc[23821](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997591, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.21.12.190@KRBTEST.COM
I20260806 06:26:31.994117 24741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:31.987130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47546 (local address 127.21.12.190:43693)
0806 06:26:31.987308 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0806 06:26:31.987310 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:26:31.987376 (+ 66us) server_negotiation.cc:407] Connection header received
0806 06:26:31.987449 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:26:31.987450 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:26:31.987462 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:26:31.987519 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:26:31.988066 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:31.988577 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:31.989284 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:31.989480 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:31.989532 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:31.991788 (+ 2256us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:26:31.991789 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:26:31.991790 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 06:26:31.991797 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:26:31.993207 (+ 1410us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:26:31.993485 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:26:31.993486 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:26:31.993486 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:26:31.993536 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:26:31.993667 (+ 131us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:26:31.993668 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:26:31.993668 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:26:31.993760 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:26:31.993828 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0806 06:26:31.993897 (+ 69us) server_negotiation.cc:299] Negotiation successful
0806 06:26:31.993956 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":96,"threads_started":1}
I20260806 06:26:31.995491 23880 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47546:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 06:26:31.995636 23880 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 06:26:31.997016 24501 tablet_service.cc:1467] Processing CreateTablet for tablet dbf068c78b894a599e98e43769dd951e (DEFAULT_TABLE table=test-table [id=a076bb389ceb441780c5487303f7966c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 06:26:31.997028 24074 tablet_service.cc:1467] Processing CreateTablet for tablet dbf068c78b894a599e98e43769dd951e (DEFAULT_TABLE table=test-table [id=a076bb389ceb441780c5487303f7966c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 06:26:31.997082 24291 tablet_service.cc:1467] Processing CreateTablet for tablet dbf068c78b894a599e98e43769dd951e (DEFAULT_TABLE table=test-table [id=a076bb389ceb441780c5487303f7966c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 06:26:31.997149 24074 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dbf068c78b894a599e98e43769dd951e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 06:26:31.997149 24501 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dbf068c78b894a599e98e43769dd951e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 06:26:31.997223 24291 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dbf068c78b894a599e98e43769dd951e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 06:26:31.998454 24744 tablet_bootstrap.cc:492] T dbf068c78b894a599e98e43769dd951e P 0db93c757c334db8946a1e5df997c5d9: Bootstrap starting.
I20260806 06:26:31.998770 24745 tablet_bootstrap.cc:492] T dbf068c78b894a599e98e43769dd951e P 9447631f55b7404f9ab0232e0a1a2dc8: Bootstrap starting.
I20260806 06:26:31.999006 24744 tablet_bootstrap.cc:654] T dbf068c78b894a599e98e43769dd951e P 0db93c757c334db8946a1e5df997c5d9: Neither blocks nor log segments found. Creating new log.
I20260806 06:26:31.999167 24745 tablet_bootstrap.cc:654] T dbf068c78b894a599e98e43769dd951e P 9447631f55b7404f9ab0232e0a1a2dc8: Neither blocks nor log segments found. Creating new log.
I20260806 06:26:31.999471 24746 tablet_bootstrap.cc:492] T dbf068c78b894a599e98e43769dd951e P 09211067c8c14e159b2941318ccdacb5: Bootstrap starting.
I20260806 06:26:31.999825 24745 tablet_bootstrap.cc:492] T dbf068c78b894a599e98e43769dd951e P 9447631f55b7404f9ab0232e0a1a2dc8: No bootstrap required, opened a new log
I20260806 06:26:31.999850 24746 tablet_bootstrap.cc:654] T dbf068c78b894a599e98e43769dd951e P 09211067c8c14e159b2941318ccdacb5: Neither blocks nor log segments found. Creating new log.
I20260806 06:26:31.999890 24745 ts_tablet_manager.cc:1397] T dbf068c78b894a599e98e43769dd951e P 9447631f55b7404f9ab0232e0a1a2dc8: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260806 06:26:32.000020 24745 raft_consensus.cc:348] T dbf068c78b894a599e98e43769dd951e P 9447631f55b7404f9ab0232e0a1a2dc8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9447631f55b7404f9ab0232e0a1a2dc8" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 46063 } } peers { permanent_uuid: "0db93c757c334db8946a1e5df997c5d9" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 43279 } } peers { permanent_uuid: "09211067c8c14e159b2941318ccdacb5" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 34181 } }
I20260806 06:26:32.000104 24745 raft_consensus.cc:374] T dbf068c78b894a599e98e43769dd951e P 9447631f55b7404f9ab0232e0a1a2dc8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 06:26:32.000136 24745 raft_consensus.cc:729] T dbf068c78b894a599e98e43769dd951e P 9447631f55b7404f9ab0232e0a1a2dc8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9447631f55b7404f9ab0232e0a1a2dc8, State: Initialized, Role: FOLLOWER
I20260806 06:26:32.000197 24745 consensus_queue.cc:260] T dbf068c78b894a599e98e43769dd951e P 9447631f55b7404f9ab0232e0a1a2dc8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9447631f55b7404f9ab0232e0a1a2dc8" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 46063 } } peers { permanent_uuid: "0db93c757c334db8946a1e5df997c5d9" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 43279 } } peers { permanent_uuid: "09211067c8c14e159b2941318ccdacb5" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 34181 } }
I20260806 06:26:32.000309 24745 ts_tablet_manager.cc:1428] T dbf068c78b894a599e98e43769dd951e P 9447631f55b7404f9ab0232e0a1a2dc8: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260806 06:26:32.000535 24746 tablet_bootstrap.cc:492] T dbf068c78b894a599e98e43769dd951e P 09211067c8c14e159b2941318ccdacb5: No bootstrap required, opened a new log
I20260806 06:26:32.000545 24744 tablet_bootstrap.cc:492] T dbf068c78b894a599e98e43769dd951e P 0db93c757c334db8946a1e5df997c5d9: No bootstrap required, opened a new log
I20260806 06:26:32.000586 24746 ts_tablet_manager.cc:1397] T dbf068c78b894a599e98e43769dd951e P 09211067c8c14e159b2941318ccdacb5: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260806 06:26:32.000593 24744 ts_tablet_manager.cc:1397] T dbf068c78b894a599e98e43769dd951e P 0db93c757c334db8946a1e5df997c5d9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260806 06:26:32.000722 24744 raft_consensus.cc:348] T dbf068c78b894a599e98e43769dd951e P 0db93c757c334db8946a1e5df997c5d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9447631f55b7404f9ab0232e0a1a2dc8" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 46063 } } peers { permanent_uuid: "0db93c757c334db8946a1e5df997c5d9" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 43279 } } peers { permanent_uuid: "09211067c8c14e159b2941318ccdacb5" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 34181 } }
I20260806 06:26:32.000798 24744 raft_consensus.cc:374] T dbf068c78b894a599e98e43769dd951e P 0db93c757c334db8946a1e5df997c5d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 06:26:32.000821 24744 raft_consensus.cc:729] T dbf068c78b894a599e98e43769dd951e P 0db93c757c334db8946a1e5df997c5d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0db93c757c334db8946a1e5df997c5d9, State: Initialized, Role: FOLLOWER
I20260806 06:26:32.000871 24744 consensus_queue.cc:260] T dbf068c78b894a599e98e43769dd951e P 0db93c757c334db8946a1e5df997c5d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9447631f55b7404f9ab0232e0a1a2dc8" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 46063 } } peers { permanent_uuid: "0db93c757c334db8946a1e5df997c5d9" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 43279 } } peers { permanent_uuid: "09211067c8c14e159b2941318ccdacb5" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 34181 } }
I20260806 06:26:32.000846 24746 raft_consensus.cc:348] T dbf068c78b894a599e98e43769dd951e P 09211067c8c14e159b2941318ccdacb5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9447631f55b7404f9ab0232e0a1a2dc8" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 46063 } } peers { permanent_uuid: "0db93c757c334db8946a1e5df997c5d9" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 43279 } } peers { permanent_uuid: "09211067c8c14e159b2941318ccdacb5" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 34181 } }
I20260806 06:26:32.000957 24746 raft_consensus.cc:374] T dbf068c78b894a599e98e43769dd951e P 09211067c8c14e159b2941318ccdacb5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 06:26:32.000989 24746 raft_consensus.cc:729] T dbf068c78b894a599e98e43769dd951e P 09211067c8c14e159b2941318ccdacb5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09211067c8c14e159b2941318ccdacb5, State: Initialized, Role: FOLLOWER
I20260806 06:26:32.001065 24744 ts_tablet_manager.cc:1428] T dbf068c78b894a599e98e43769dd951e P 0db93c757c334db8946a1e5df997c5d9: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260806 06:26:32.001045 24746 consensus_queue.cc:260] T dbf068c78b894a599e98e43769dd951e P 09211067c8c14e159b2941318ccdacb5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9447631f55b7404f9ab0232e0a1a2dc8" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 46063 } } peers { permanent_uuid: "0db93c757c334db8946a1e5df997c5d9" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 43279 } } peers { permanent_uuid: "09211067c8c14e159b2941318ccdacb5" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 34181 } }
I20260806 06:26:32.001341 24746 ts_tablet_manager.cc:1428] T dbf068c78b894a599e98e43769dd951e P 09211067c8c14e159b2941318ccdacb5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 06:26:32.081372 24752 raft_consensus.cc:482] T dbf068c78b894a599e98e43769dd951e P 0db93c757c334db8946a1e5df997c5d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 06:26:32.081466 24752 raft_consensus.cc:504] T dbf068c78b894a599e98e43769dd951e P 0db93c757c334db8946a1e5df997c5d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9447631f55b7404f9ab0232e0a1a2dc8" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 46063 } } peers { permanent_uuid: "0db93c757c334db8946a1e5df997c5d9" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 43279 } } peers { permanent_uuid: "09211067c8c14e159b2941318ccdacb5" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 34181 } }
I20260806 06:26:32.081813 24752 leader_election.cc:288] T dbf068c78b894a599e98e43769dd951e P 0db93c757c334db8946a1e5df997c5d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9447631f55b7404f9ab0232e0a1a2dc8 (127.21.12.129:46063), 09211067c8c14e159b2941318ccdacb5 (127.21.12.131:34181)
I20260806 06:26:32.084826 24756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:32.081984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.130:35545 (local address 127.21.12.129:46063)
0806 06:26:32.082271 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0806 06:26:32.082273 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:26:32.082283 (+ 10us) server_negotiation.cc:407] Connection header received
0806 06:26:32.082364 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:26:32.082365 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:26:32.082377 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:26:32.082456 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 06:26:32.082634 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:32.083114 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:32.084138 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:32.084492 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:32.084553 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:32.084563 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0806 06:26:32.084708 (+ 145us) server_negotiation.cc:299] Negotiation successful
0806 06:26:32.084748 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":235,"thread_start_us":169,"threads_started":1}
I20260806 06:26:32.084817 24758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:32.082058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.130:42119 (local address 127.21.12.131:34181)
0806 06:26:32.082209 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0806 06:26:32.082211 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:26:32.082336 (+ 125us) server_negotiation.cc:407] Connection header received
0806 06:26:32.082408 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:26:32.082408 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:26:32.082417 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:26:32.082478 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 06:26:32.082619 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:32.083114 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:32.084129 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:32.084503 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:32.084567 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:32.084575 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0806 06:26:32.084685 (+ 110us) server_negotiation.cc:299] Negotiation successful
0806 06:26:32.084725 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":89,"threads_started":1}
I20260806 06:26:32.084817 24754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:32.081971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.131:34181 (local address 127.21.12.130:42119)
0806 06:26:32.082247 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0806 06:26:32.082258 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0806 06:26:32.082347 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:26:32.082487 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:26:32.082488 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:26:32.082490 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 06:26:32.082558 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:32.082560 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:32.083185 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:32.083186 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:32.084044 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:32.084046 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:32.084577 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:32.084578 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:32.084627 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:32.084630 (+ 3us) client_negotiation.cc:769] Sending connection context
0806 06:26:32.084669 (+ 39us) client_negotiation.cc:240] Negotiation successful
0806 06:26:32.084707 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"mutex_wait_us":1,"thread_start_us":187,"threads_started":1}
I20260806 06:26:32.084841 24757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:32.081940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.129:46063 (local address 127.21.12.130:35545)
0806 06:26:32.082207 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0806 06:26:32.082218 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0806 06:26:32.082275 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:26:32.082471 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:26:32.082472 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:26:32.082475 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 06:26:32.082564 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:32.082566 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:32.083189 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:32.083190 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:32.084047 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:32.084049 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:32.084569 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:32.084570 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:32.084642 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:32.084645 (+ 3us) client_negotiation.cc:769] Sending connection context
0806 06:26:32.084695 (+ 50us) client_negotiation.cc:240] Negotiation successful
0806 06:26:32.084740 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"mutex_wait_us":37,"thread_start_us":140,"threads_started":1}
I20260806 06:26:32.085067 24527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbf068c78b894a599e98e43769dd951e" candidate_uuid: "0db93c757c334db8946a1e5df997c5d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09211067c8c14e159b2941318ccdacb5" is_pre_election: true
I20260806 06:26:32.085163 24527 raft_consensus.cc:2454] T dbf068c78b894a599e98e43769dd951e P 09211067c8c14e159b2941318ccdacb5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0db93c757c334db8946a1e5df997c5d9 in term 0.
I20260806 06:26:32.085275 24105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbf068c78b894a599e98e43769dd951e" candidate_uuid: "0db93c757c334db8946a1e5df997c5d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9447631f55b7404f9ab0232e0a1a2dc8" is_pre_election: true
I20260806 06:26:32.085358 24213 leader_election.cc:302] T dbf068c78b894a599e98e43769dd951e P 0db93c757c334db8946a1e5df997c5d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09211067c8c14e159b2941318ccdacb5, 0db93c757c334db8946a1e5df997c5d9; no voters:
I20260806 06:26:32.085425 24105 raft_consensus.cc:2454] T dbf068c78b894a599e98e43769dd951e P 9447631f55b7404f9ab0232e0a1a2dc8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0db93c757c334db8946a1e5df997c5d9 in term 0.
I20260806 06:26:32.085472 24752 raft_consensus.cc:2781] T dbf068c78b894a599e98e43769dd951e P 0db93c757c334db8946a1e5df997c5d9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 06:26:32.085503 24752 raft_consensus.cc:482] T dbf068c78b894a599e98e43769dd951e P 0db93c757c334db8946a1e5df997c5d9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 06:26:32.085521 24752 raft_consensus.cc:3037] T dbf068c78b894a599e98e43769dd951e P 0db93c757c334db8946a1e5df997c5d9 [term 0 FOLLOWER]: Advancing to term 1
I20260806 06:26:32.086256 24752 raft_consensus.cc:504] T dbf068c78b894a599e98e43769dd951e P 0db93c757c334db8946a1e5df997c5d9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9447631f55b7404f9ab0232e0a1a2dc8" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 46063 } } peers { permanent_uuid: "0db93c757c334db8946a1e5df997c5d9" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 43279 } } peers { permanent_uuid: "09211067c8c14e159b2941318ccdacb5" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 34181 } }
I20260806 06:26:32.086413 24752 leader_election.cc:288] T dbf068c78b894a599e98e43769dd951e P 0db93c757c334db8946a1e5df997c5d9 [CANDIDATE]: Term 1 election: Requested vote from peers 9447631f55b7404f9ab0232e0a1a2dc8 (127.21.12.129:46063), 09211067c8c14e159b2941318ccdacb5 (127.21.12.131:34181)
I20260806 06:26:32.086575 24527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbf068c78b894a599e98e43769dd951e" candidate_uuid: "0db93c757c334db8946a1e5df997c5d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09211067c8c14e159b2941318ccdacb5"
I20260806 06:26:32.086652 24527 raft_consensus.cc:3037] T dbf068c78b894a599e98e43769dd951e P 09211067c8c14e159b2941318ccdacb5 [term 0 FOLLOWER]: Advancing to term 1
I20260806 06:26:32.087340 24527 raft_consensus.cc:2454] T dbf068c78b894a599e98e43769dd951e P 09211067c8c14e159b2941318ccdacb5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0db93c757c334db8946a1e5df997c5d9 in term 1.
I20260806 06:26:32.087442 24105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbf068c78b894a599e98e43769dd951e" candidate_uuid: "0db93c757c334db8946a1e5df997c5d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9447631f55b7404f9ab0232e0a1a2dc8"
I20260806 06:26:32.087525 24105 raft_consensus.cc:3037] T dbf068c78b894a599e98e43769dd951e P 9447631f55b7404f9ab0232e0a1a2dc8 [term 0 FOLLOWER]: Advancing to term 1
I20260806 06:26:32.087496 24213 leader_election.cc:302] T dbf068c78b894a599e98e43769dd951e P 0db93c757c334db8946a1e5df997c5d9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09211067c8c14e159b2941318ccdacb5, 0db93c757c334db8946a1e5df997c5d9; no voters:
I20260806 06:26:32.087605 24752 raft_consensus.cc:2781] T dbf068c78b894a599e98e43769dd951e P 0db93c757c334db8946a1e5df997c5d9 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 06:26:32.087755 24752 raft_consensus.cc:686] T dbf068c78b894a599e98e43769dd951e P 0db93c757c334db8946a1e5df997c5d9 [term 1 LEADER]: Becoming Leader. State: Replica: 0db93c757c334db8946a1e5df997c5d9, State: Running, Role: LEADER
I20260806 06:26:32.087826 24752 consensus_queue.cc:237] T dbf068c78b894a599e98e43769dd951e P 0db93c757c334db8946a1e5df997c5d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9447631f55b7404f9ab0232e0a1a2dc8" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 46063 } } peers { permanent_uuid: "0db93c757c334db8946a1e5df997c5d9" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 43279 } } peers { permanent_uuid: "09211067c8c14e159b2941318ccdacb5" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 34181 } }
I20260806 06:26:32.088229 24105 raft_consensus.cc:2454] T dbf068c78b894a599e98e43769dd951e P 9447631f55b7404f9ab0232e0a1a2dc8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0db93c757c334db8946a1e5df997c5d9 in term 1.
I20260806 06:26:32.088486 23881 catalog_manager.cc:5495] T dbf068c78b894a599e98e43769dd951e P 0db93c757c334db8946a1e5df997c5d9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0db93c757c334db8946a1e5df997c5d9 (127.21.12.130). New cstate: current_term: 1 leader_uuid: "0db93c757c334db8946a1e5df997c5d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9447631f55b7404f9ab0232e0a1a2dc8" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 46063 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0db93c757c334db8946a1e5df997c5d9" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 43279 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "09211067c8c14e159b2941318ccdacb5" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 34181 } health_report { overall_health: UNKNOWN } } }
I20260806 06:26:32.094673 24763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:32.092309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44380 (local address 127.21.12.130:43279)
0806 06:26:32.092448 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0806 06:26:32.092450 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:26:32.092458 (+ 8us) server_negotiation.cc:407] Connection header received
0806 06:26:32.092487 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:26:32.092488 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:26:32.092529 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:26:32.092597 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 06:26:32.092720 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:32.093281 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:32.093942 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:32.094167 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:32.094212 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:32.094366 (+ 154us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 06:26:32.094468 (+ 102us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 06:26:32.094547 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0806 06:26:32.094577 (+ 30us) server_negotiation.cc:299] Negotiation successful
0806 06:26:32.094596 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":90,"threads_started":1}
I20260806 06:26:32.099138 24756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:32.096839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.130:54889 (local address 127.21.12.129:46063)
0806 06:26:32.096895 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0806 06:26:32.096897 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:26:32.097020 (+ 123us) server_negotiation.cc:407] Connection header received
0806 06:26:32.097096 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:26:32.097097 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:26:32.097106 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:26:32.097155 (+ 49us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 06:26:32.097276 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:32.097774 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:32.098575 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:32.098768 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:32.098831 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:32.098957 (+ 126us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 06:26:32.098990 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 06:26:32.099033 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0806 06:26:32.099068 (+ 35us) server_negotiation.cc:299] Negotiation successful
0806 06:26:32.099085 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260806 06:26:32.099156 24769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:32.096777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.129:46063 (local address 127.21.12.130:54889)
0806 06:26:32.096954 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0806 06:26:32.096965 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0806 06:26:32.097039 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:26:32.097165 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:26:32.097166 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:26:32.097167 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 06:26:32.097236 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:32.097237 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:32.097836 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:32.097837 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:32.098485 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:32.098487 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:32.098840 (+ 353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:32.098841 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:32.098888 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:32.098900 (+ 12us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 06:26:32.099034 (+ 134us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 06:26:32.099035 (+ 1us) client_negotiation.cc:769] Sending connection context
0806 06:26:32.099065 (+ 30us) client_negotiation.cc:240] Negotiation successful
0806 06:26:32.099097 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":82,"threads_started":1}
I20260806 06:26:32.101125 24527 raft_consensus.cc:1261] T dbf068c78b894a599e98e43769dd951e P 09211067c8c14e159b2941318ccdacb5 [term 1 FOLLOWER]: Refusing update from remote peer 0db93c757c334db8946a1e5df997c5d9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 06:26:32.101238 24105 raft_consensus.cc:1261] T dbf068c78b894a599e98e43769dd951e P 9447631f55b7404f9ab0232e0a1a2dc8 [term 1 FOLLOWER]: Refusing update from remote peer 0db93c757c334db8946a1e5df997c5d9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 06:26:32.101392 24759 consensus_queue.cc:1035] T dbf068c78b894a599e98e43769dd951e P 0db93c757c334db8946a1e5df997c5d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09211067c8c14e159b2941318ccdacb5" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 34181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 06:26:32.101505 24752 consensus_queue.cc:1035] T dbf068c78b894a599e98e43769dd951e P 0db93c757c334db8946a1e5df997c5d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9447631f55b7404f9ab0232e0a1a2dc8" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 46063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 06:26:32.108569 24763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:32.106318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.129:54077 (local address 127.21.12.130:43279)
0806 06:26:32.106355 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0806 06:26:32.106356 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0806 06:26:32.106494 (+ 138us) server_negotiation.cc:407] Connection header received
0806 06:26:32.106556 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:26:32.106557 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:26:32.106565 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:26:32.106608 (+ 43us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 06:26:32.106694 (+ 86us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:32.107133 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:32.107843 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:32.108085 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:32.108148 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:32.108361 (+ 213us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 06:26:32.108384 (+ 23us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 06:26:32.108422 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0806 06:26:32.108480 (+ 58us) server_negotiation.cc:299] Negotiation successful
0806 06:26:32.108498 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260806 06:26:32.108592 24774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:32.106210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.130:43279 (local address 127.21.12.129:54077)
0806 06:26:32.106429 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0806 06:26:32.106439 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0806 06:26:32.106494 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:26:32.106610 (+ 116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:26:32.106610 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:26:32.106612 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 06:26:32.106651 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:32.106653 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:32.107195 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:32.107196 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:32.107781 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:32.107783 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:32.108157 (+ 374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:32.108158 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:32.108278 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:32.108284 (+ 6us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 06:26:32.108434 (+ 150us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 06:26:32.108437 (+ 3us) client_negotiation.cc:769] Sending connection context
0806 06:26:32.108479 (+ 42us) client_negotiation.cc:240] Negotiation successful
0806 06:26:32.108523 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":137,"threads_started":1}
I20260806 06:26:32.114591 23880 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47546:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 06:26:32.114677 23880 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47546:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 06:26:32.115267 23880 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b3579394671149a89ff9b5c9f4b9854e: Sending DeleteTablet for 3 replicas of tablet dbf068c78b894a599e98e43769dd951e
I20260806 06:26:32.115533 24501 tablet_service.cc:1514] Processing DeleteTablet for tablet dbf068c78b894a599e98e43769dd951e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 06:26:32 UTC) from {username='kudu', principal='kudu/127.21.12.190@KRBTEST.COM'} at 127.0.0.1:40218
I20260806 06:26:32.115605 24074 tablet_service.cc:1514] Processing DeleteTablet for tablet dbf068c78b894a599e98e43769dd951e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 06:26:32 UTC) from {username='kudu', principal='kudu/127.21.12.190@KRBTEST.COM'} at 127.0.0.1:45130
I20260806 06:26:32.115636 24734 tablet_replica.cc:330] T dbf068c78b894a599e98e43769dd951e P 09211067c8c14e159b2941318ccdacb5: stopping tablet replica
I20260806 06:26:32.115679 24734 raft_consensus.cc:2229] T dbf068c78b894a599e98e43769dd951e P 09211067c8c14e159b2941318ccdacb5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 06:26:32.115690 24732 tablet_replica.cc:330] T dbf068c78b894a599e98e43769dd951e P 9447631f55b7404f9ab0232e0a1a2dc8: stopping tablet replica
I20260806 06:26:32.115715 24734 raft_consensus.cc:2258] T dbf068c78b894a599e98e43769dd951e P 09211067c8c14e159b2941318ccdacb5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 06:26:32.115754 24732 raft_consensus.cc:2229] T dbf068c78b894a599e98e43769dd951e P 9447631f55b7404f9ab0232e0a1a2dc8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 06:26:32.115787 24732 raft_consensus.cc:2258] T dbf068c78b894a599e98e43769dd951e P 9447631f55b7404f9ab0232e0a1a2dc8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 06:26:32.115778 24291 tablet_service.cc:1514] Processing DeleteTablet for tablet dbf068c78b894a599e98e43769dd951e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 06:26:32 UTC) from {username='kudu', principal='kudu/127.21.12.190@KRBTEST.COM'} at 127.0.0.1:44364
I20260806 06:26:32.115849 24730 tablet_replica.cc:330] T dbf068c78b894a599e98e43769dd951e P 0db93c757c334db8946a1e5df997c5d9: stopping tablet replica
I20260806 06:26:32.115890 24730 raft_consensus.cc:2229] T dbf068c78b894a599e98e43769dd951e P 0db93c757c334db8946a1e5df997c5d9 [term 1 LEADER]: Raft consensus shutting down.
I20260806 06:26:32.115959 24730 raft_consensus.cc:2258] T dbf068c78b894a599e98e43769dd951e P 0db93c757c334db8946a1e5df997c5d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 06:26:32.116110 24732 ts_tablet_manager.cc:1905] T dbf068c78b894a599e98e43769dd951e P 9447631f55b7404f9ab0232e0a1a2dc8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 06:26:32.116345 24734 ts_tablet_manager.cc:1905] T dbf068c78b894a599e98e43769dd951e P 09211067c8c14e159b2941318ccdacb5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 06:26:32.117489 23880 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47546:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 06:26:32.117628 23880 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 06:26:32.117733 24732 ts_tablet_manager.cc:1918] T dbf068c78b894a599e98e43769dd951e P 9447631f55b7404f9ab0232e0a1a2dc8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 06:26:32.117796 24732 log.cc:1196] T dbf068c78b894a599e98e43769dd951e P 9447631f55b7404f9ab0232e0a1a2dc8: Deleting WAL directory at /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785997581049705-21554-0/minicluster-data/ts-0/wal/wals/dbf068c78b894a599e98e43769dd951e
I20260806 06:26:32.118001 24734 ts_tablet_manager.cc:1918] T dbf068c78b894a599e98e43769dd951e P 09211067c8c14e159b2941318ccdacb5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 06:26:32.118009 24732 ts_tablet_manager.cc:1939] T dbf068c78b894a599e98e43769dd951e P 9447631f55b7404f9ab0232e0a1a2dc8: Deleting consensus metadata
I20260806 06:26:32.118057 24734 log.cc:1196] T dbf068c78b894a599e98e43769dd951e P 09211067c8c14e159b2941318ccdacb5: Deleting WAL directory at /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785997581049705-21554-0/minicluster-data/ts-2/wal/wals/dbf068c78b894a599e98e43769dd951e
I20260806 06:26:32.118251 24734 ts_tablet_manager.cc:1939] T dbf068c78b894a599e98e43769dd951e P 09211067c8c14e159b2941318ccdacb5: Deleting consensus metadata
I20260806 06:26:32.118346 23865 catalog_manager.cc:4841] TS 9447631f55b7404f9ab0232e0a1a2dc8 (127.21.12.129:46063): tablet dbf068c78b894a599e98e43769dd951e (table test-table [id=a076bb389ceb441780c5487303f7966c]) successfully deleted
I20260806 06:26:32.118427 24730 ts_tablet_manager.cc:1905] T dbf068c78b894a599e98e43769dd951e P 0db93c757c334db8946a1e5df997c5d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 06:26:32.118546 23862 catalog_manager.cc:4841] TS 09211067c8c14e159b2941318ccdacb5 (127.21.12.131:34181): tablet dbf068c78b894a599e98e43769dd951e (table test-table [id=a076bb389ceb441780c5487303f7966c]) successfully deleted
I20260806 06:26:32.119351 24074 tablet_service.cc:1467] Processing CreateTablet for tablet 1edf597828e94a6787aa7aa0231080d0 (DEFAULT_TABLE table=test-table [id=ecc76554ccb440b2992fc5bdd27106cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 06:26:32.119380 24501 tablet_service.cc:1467] Processing CreateTablet for tablet 1edf597828e94a6787aa7aa0231080d0 (DEFAULT_TABLE table=test-table [id=ecc76554ccb440b2992fc5bdd27106cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 06:26:32.119455 24074 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1edf597828e94a6787aa7aa0231080d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 06:26:32.119455 24501 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1edf597828e94a6787aa7aa0231080d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 06:26:32.119940 24730 ts_tablet_manager.cc:1918] T dbf068c78b894a599e98e43769dd951e P 0db93c757c334db8946a1e5df997c5d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 06:26:32.119987 24730 log.cc:1196] T dbf068c78b894a599e98e43769dd951e P 0db93c757c334db8946a1e5df997c5d9: Deleting WAL directory at /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785997581049705-21554-0/minicluster-data/ts-1/wal/wals/dbf068c78b894a599e98e43769dd951e
I20260806 06:26:32.120198 24730 ts_tablet_manager.cc:1939] T dbf068c78b894a599e98e43769dd951e P 0db93c757c334db8946a1e5df997c5d9: Deleting consensus metadata
I20260806 06:26:32.120620 24746 tablet_bootstrap.cc:492] T 1edf597828e94a6787aa7aa0231080d0 P 09211067c8c14e159b2941318ccdacb5: Bootstrap starting.
I20260806 06:26:32.120769 24291 tablet_service.cc:1467] Processing CreateTablet for tablet 1edf597828e94a6787aa7aa0231080d0 (DEFAULT_TABLE table=test-table [id=ecc76554ccb440b2992fc5bdd27106cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 06:26:32.120817 24745 tablet_bootstrap.cc:492] T 1edf597828e94a6787aa7aa0231080d0 P 9447631f55b7404f9ab0232e0a1a2dc8: Bootstrap starting.
I20260806 06:26:32.120859 24291 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1edf597828e94a6787aa7aa0231080d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 06:26:32.120967 24746 tablet_bootstrap.cc:654] T 1edf597828e94a6787aa7aa0231080d0 P 09211067c8c14e159b2941318ccdacb5: Neither blocks nor log segments found. Creating new log.
I20260806 06:26:32.121078 24745 tablet_bootstrap.cc:654] T 1edf597828e94a6787aa7aa0231080d0 P 9447631f55b7404f9ab0232e0a1a2dc8: Neither blocks nor log segments found. Creating new log.
I20260806 06:26:32.121958 24744 tablet_bootstrap.cc:492] T 1edf597828e94a6787aa7aa0231080d0 P 0db93c757c334db8946a1e5df997c5d9: Bootstrap starting.
I20260806 06:26:32.121990 24746 tablet_bootstrap.cc:492] T 1edf597828e94a6787aa7aa0231080d0 P 09211067c8c14e159b2941318ccdacb5: No bootstrap required, opened a new log
I20260806 06:26:32.122020 24745 tablet_bootstrap.cc:492] T 1edf597828e94a6787aa7aa0231080d0 P 9447631f55b7404f9ab0232e0a1a2dc8: No bootstrap required, opened a new log
I20260806 06:26:32.122025 24746 ts_tablet_manager.cc:1397] T 1edf597828e94a6787aa7aa0231080d0 P 09211067c8c14e159b2941318ccdacb5: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260806 06:26:32.122066 24745 ts_tablet_manager.cc:1397] T 1edf597828e94a6787aa7aa0231080d0 P 9447631f55b7404f9ab0232e0a1a2dc8: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260806 06:26:32.122164 24745 raft_consensus.cc:348] T 1edf597828e94a6787aa7aa0231080d0 P 9447631f55b7404f9ab0232e0a1a2dc8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9447631f55b7404f9ab0232e0a1a2dc8" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 46063 } } peers { permanent_uuid: "0db93c757c334db8946a1e5df997c5d9" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 43279 } } peers { permanent_uuid: "09211067c8c14e159b2941318ccdacb5" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 34181 } }
I20260806 06:26:32.122226 24745 raft_consensus.cc:374] T 1edf597828e94a6787aa7aa0231080d0 P 9447631f55b7404f9ab0232e0a1a2dc8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 06:26:32.122247 24745 raft_consensus.cc:729] T 1edf597828e94a6787aa7aa0231080d0 P 9447631f55b7404f9ab0232e0a1a2dc8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9447631f55b7404f9ab0232e0a1a2dc8, State: Initialized, Role: FOLLOWER
I20260806 06:26:32.122310 24745 consensus_queue.cc:260] T 1edf597828e94a6787aa7aa0231080d0 P 9447631f55b7404f9ab0232e0a1a2dc8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9447631f55b7404f9ab0232e0a1a2dc8" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 46063 } } peers { permanent_uuid: "0db93c757c334db8946a1e5df997c5d9" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 43279 } } peers { permanent_uuid: "09211067c8c14e159b2941318ccdacb5" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 34181 } }
I20260806 06:26:32.122363 24744 tablet_bootstrap.cc:654] T 1edf597828e94a6787aa7aa0231080d0 P 0db93c757c334db8946a1e5df997c5d9: Neither blocks nor log segments found. Creating new log.
I20260806 06:26:32.122401 24745 ts_tablet_manager.cc:1428] T 1edf597828e94a6787aa7aa0231080d0 P 9447631f55b7404f9ab0232e0a1a2dc8: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260806 06:26:32.122485 24746 raft_consensus.cc:348] T 1edf597828e94a6787aa7aa0231080d0 P 09211067c8c14e159b2941318ccdacb5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9447631f55b7404f9ab0232e0a1a2dc8" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 46063 } } peers { permanent_uuid: "0db93c757c334db8946a1e5df997c5d9" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 43279 } } peers { permanent_uuid: "09211067c8c14e159b2941318ccdacb5" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 34181 } }
I20260806 06:26:32.122555 24746 raft_consensus.cc:374] T 1edf597828e94a6787aa7aa0231080d0 P 09211067c8c14e159b2941318ccdacb5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 06:26:32.122577 24746 raft_consensus.cc:729] T 1edf597828e94a6787aa7aa0231080d0 P 09211067c8c14e159b2941318ccdacb5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09211067c8c14e159b2941318ccdacb5, State: Initialized, Role: FOLLOWER
I20260806 06:26:32.122618 24746 consensus_queue.cc:260] T 1edf597828e94a6787aa7aa0231080d0 P 09211067c8c14e159b2941318ccdacb5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9447631f55b7404f9ab0232e0a1a2dc8" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 46063 } } peers { permanent_uuid: "0db93c757c334db8946a1e5df997c5d9" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 43279 } } peers { permanent_uuid: "09211067c8c14e159b2941318ccdacb5" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 34181 } }
I20260806 06:26:32.122696 24746 ts_tablet_manager.cc:1428] T 1edf597828e94a6787aa7aa0231080d0 P 09211067c8c14e159b2941318ccdacb5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 06:26:32.122758 23862 catalog_manager.cc:4841] TS 0db93c757c334db8946a1e5df997c5d9 (127.21.12.130:43279): tablet dbf068c78b894a599e98e43769dd951e (table test-table [id=a076bb389ceb441780c5487303f7966c]) successfully deleted
I20260806 06:26:32.123207 24744 tablet_bootstrap.cc:492] T 1edf597828e94a6787aa7aa0231080d0 P 0db93c757c334db8946a1e5df997c5d9: No bootstrap required, opened a new log
I20260806 06:26:32.123255 24744 ts_tablet_manager.cc:1397] T 1edf597828e94a6787aa7aa0231080d0 P 0db93c757c334db8946a1e5df997c5d9: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260806 06:26:32.123409 24744 raft_consensus.cc:348] T 1edf597828e94a6787aa7aa0231080d0 P 0db93c757c334db8946a1e5df997c5d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9447631f55b7404f9ab0232e0a1a2dc8" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 46063 } } peers { permanent_uuid: "0db93c757c334db8946a1e5df997c5d9" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 43279 } } peers { permanent_uuid: "09211067c8c14e159b2941318ccdacb5" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 34181 } }
I20260806 06:26:32.123472 24744 raft_consensus.cc:374] T 1edf597828e94a6787aa7aa0231080d0 P 0db93c757c334db8946a1e5df997c5d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 06:26:32.123492 24744 raft_consensus.cc:729] T 1edf597828e94a6787aa7aa0231080d0 P 0db93c757c334db8946a1e5df997c5d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0db93c757c334db8946a1e5df997c5d9, State: Initialized, Role: FOLLOWER
I20260806 06:26:32.123528 24744 consensus_queue.cc:260] T 1edf597828e94a6787aa7aa0231080d0 P 0db93c757c334db8946a1e5df997c5d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9447631f55b7404f9ab0232e0a1a2dc8" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 46063 } } peers { permanent_uuid: "0db93c757c334db8946a1e5df997c5d9" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 43279 } } peers { permanent_uuid: "09211067c8c14e159b2941318ccdacb5" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 34181 } }
I20260806 06:26:32.123656 24744 ts_tablet_manager.cc:1428] T 1edf597828e94a6787aa7aa0231080d0 P 0db93c757c334db8946a1e5df997c5d9: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
W20260806 06:26:32.140828 24182 tablet.cc:2369] T 1edf597828e94a6787aa7aa0231080d0 P 9447631f55b7404f9ab0232e0a1a2dc8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 06:26:32.185009 24759 raft_consensus.cc:482] T 1edf597828e94a6787aa7aa0231080d0 P 0db93c757c334db8946a1e5df997c5d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 06:26:32.185132 24759 raft_consensus.cc:504] T 1edf597828e94a6787aa7aa0231080d0 P 0db93c757c334db8946a1e5df997c5d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9447631f55b7404f9ab0232e0a1a2dc8" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 46063 } } peers { permanent_uuid: "0db93c757c334db8946a1e5df997c5d9" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 43279 } } peers { permanent_uuid: "09211067c8c14e159b2941318ccdacb5" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 34181 } }
I20260806 06:26:32.185313 24759 leader_election.cc:288] T 1edf597828e94a6787aa7aa0231080d0 P 0db93c757c334db8946a1e5df997c5d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9447631f55b7404f9ab0232e0a1a2dc8 (127.21.12.129:46063), 09211067c8c14e159b2941318ccdacb5 (127.21.12.131:34181)
I20260806 06:26:32.185510 24105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1edf597828e94a6787aa7aa0231080d0" candidate_uuid: "0db93c757c334db8946a1e5df997c5d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9447631f55b7404f9ab0232e0a1a2dc8" is_pre_election: true
I20260806 06:26:32.185621 24105 raft_consensus.cc:2454] T 1edf597828e94a6787aa7aa0231080d0 P 9447631f55b7404f9ab0232e0a1a2dc8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0db93c757c334db8946a1e5df997c5d9 in term 0.
I20260806 06:26:32.185530 24527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1edf597828e94a6787aa7aa0231080d0" candidate_uuid: "0db93c757c334db8946a1e5df997c5d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09211067c8c14e159b2941318ccdacb5" is_pre_election: true
I20260806 06:26:32.185672 24527 raft_consensus.cc:2454] T 1edf597828e94a6787aa7aa0231080d0 P 09211067c8c14e159b2941318ccdacb5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0db93c757c334db8946a1e5df997c5d9 in term 0.
I20260806 06:26:32.185786 24216 leader_election.cc:302] T 1edf597828e94a6787aa7aa0231080d0 P 0db93c757c334db8946a1e5df997c5d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0db93c757c334db8946a1e5df997c5d9, 9447631f55b7404f9ab0232e0a1a2dc8; no voters:
I20260806 06:26:32.185875 24759 raft_consensus.cc:2781] T 1edf597828e94a6787aa7aa0231080d0 P 0db93c757c334db8946a1e5df997c5d9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 06:26:32.185909 24759 raft_consensus.cc:482] T 1edf597828e94a6787aa7aa0231080d0 P 0db93c757c334db8946a1e5df997c5d9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 06:26:32.185935 24759 raft_consensus.cc:3037] T 1edf597828e94a6787aa7aa0231080d0 P 0db93c757c334db8946a1e5df997c5d9 [term 0 FOLLOWER]: Advancing to term 1
I20260806 06:26:32.186919 24759 raft_consensus.cc:504] T 1edf597828e94a6787aa7aa0231080d0 P 0db93c757c334db8946a1e5df997c5d9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9447631f55b7404f9ab0232e0a1a2dc8" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 46063 } } peers { permanent_uuid: "0db93c757c334db8946a1e5df997c5d9" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 43279 } } peers { permanent_uuid: "09211067c8c14e159b2941318ccdacb5" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 34181 } }
I20260806 06:26:32.187060 24759 leader_election.cc:288] T 1edf597828e94a6787aa7aa0231080d0 P 0db93c757c334db8946a1e5df997c5d9 [CANDIDATE]: Term 1 election: Requested vote from peers 9447631f55b7404f9ab0232e0a1a2dc8 (127.21.12.129:46063), 09211067c8c14e159b2941318ccdacb5 (127.21.12.131:34181)
I20260806 06:26:32.187268 24527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1edf597828e94a6787aa7aa0231080d0" candidate_uuid: "0db93c757c334db8946a1e5df997c5d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09211067c8c14e159b2941318ccdacb5"
I20260806 06:26:32.187288 24105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1edf597828e94a6787aa7aa0231080d0" candidate_uuid: "0db93c757c334db8946a1e5df997c5d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9447631f55b7404f9ab0232e0a1a2dc8"
I20260806 06:26:32.187346 24527 raft_consensus.cc:3037] T 1edf597828e94a6787aa7aa0231080d0 P 09211067c8c14e159b2941318ccdacb5 [term 0 FOLLOWER]: Advancing to term 1
I20260806 06:26:32.187376 24105 raft_consensus.cc:3037] T 1edf597828e94a6787aa7aa0231080d0 P 9447631f55b7404f9ab0232e0a1a2dc8 [term 0 FOLLOWER]: Advancing to term 1
I20260806 06:26:32.188073 24527 raft_consensus.cc:2454] T 1edf597828e94a6787aa7aa0231080d0 P 09211067c8c14e159b2941318ccdacb5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0db93c757c334db8946a1e5df997c5d9 in term 1.
I20260806 06:26:32.188196 24105 raft_consensus.cc:2454] T 1edf597828e94a6787aa7aa0231080d0 P 9447631f55b7404f9ab0232e0a1a2dc8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0db93c757c334db8946a1e5df997c5d9 in term 1.
I20260806 06:26:32.188256 24213 leader_election.cc:302] T 1edf597828e94a6787aa7aa0231080d0 P 0db93c757c334db8946a1e5df997c5d9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09211067c8c14e159b2941318ccdacb5, 0db93c757c334db8946a1e5df997c5d9; no voters:
I20260806 06:26:32.188346 24759 raft_consensus.cc:2781] T 1edf597828e94a6787aa7aa0231080d0 P 0db93c757c334db8946a1e5df997c5d9 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 06:26:32.188390 24759 raft_consensus.cc:686] T 1edf597828e94a6787aa7aa0231080d0 P 0db93c757c334db8946a1e5df997c5d9 [term 1 LEADER]: Becoming Leader. State: Replica: 0db93c757c334db8946a1e5df997c5d9, State: Running, Role: LEADER
I20260806 06:26:32.188432 24759 consensus_queue.cc:237] T 1edf597828e94a6787aa7aa0231080d0 P 0db93c757c334db8946a1e5df997c5d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9447631f55b7404f9ab0232e0a1a2dc8" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 46063 } } peers { permanent_uuid: "0db93c757c334db8946a1e5df997c5d9" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 43279 } } peers { permanent_uuid: "09211067c8c14e159b2941318ccdacb5" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 34181 } }
I20260806 06:26:32.189046 23880 catalog_manager.cc:5495] T 1edf597828e94a6787aa7aa0231080d0 P 0db93c757c334db8946a1e5df997c5d9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0db93c757c334db8946a1e5df997c5d9 (127.21.12.130). New cstate: current_term: 1 leader_uuid: "0db93c757c334db8946a1e5df997c5d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9447631f55b7404f9ab0232e0a1a2dc8" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 46063 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0db93c757c334db8946a1e5df997c5d9" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 43279 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "09211067c8c14e159b2941318ccdacb5" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 34181 } health_report { overall_health: UNKNOWN } } }
I20260806 06:26:32.200881 24105 raft_consensus.cc:1261] T 1edf597828e94a6787aa7aa0231080d0 P 9447631f55b7404f9ab0232e0a1a2dc8 [term 1 FOLLOWER]: Refusing update from remote peer 0db93c757c334db8946a1e5df997c5d9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 06:26:32.200881 24527 raft_consensus.cc:1261] T 1edf597828e94a6787aa7aa0231080d0 P 09211067c8c14e159b2941318ccdacb5 [term 1 FOLLOWER]: Refusing update from remote peer 0db93c757c334db8946a1e5df997c5d9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 06:26:32.201131 24759 consensus_queue.cc:1035] T 1edf597828e94a6787aa7aa0231080d0 P 0db93c757c334db8946a1e5df997c5d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9447631f55b7404f9ab0232e0a1a2dc8" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 46063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 06:26:32.201485 24772 consensus_queue.cc:1035] T 1edf597828e94a6787aa7aa0231080d0 P 0db93c757c334db8946a1e5df997c5d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09211067c8c14e159b2941318ccdacb5" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 34181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260806 06:26:32.208914 24710 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260806 06:26:33.726598 23880 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47546:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 06:26:33.726712 23880 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47546:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 06:26:33.727591 23880 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b3579394671149a89ff9b5c9f4b9854e: Sending DeleteTablet for 3 replicas of tablet 1edf597828e94a6787aa7aa0231080d0
I20260806 06:26:33.727896 24074 tablet_service.cc:1514] Processing DeleteTablet for tablet 1edf597828e94a6787aa7aa0231080d0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 06:26:33 UTC) from {username='kudu', principal='kudu/127.21.12.190@KRBTEST.COM'} at 127.0.0.1:45130
I20260806 06:26:33.727959 21554 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHhLhT/build/release/bin/kudu with pid 23986
I20260806 06:26:33.728431 24291 tablet_service.cc:1514] Processing DeleteTablet for tablet 1edf597828e94a6787aa7aa0231080d0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 06:26:33 UTC) from {username='kudu', principal='kudu/127.21.12.190@KRBTEST.COM'} at 127.0.0.1:44364
I20260806 06:26:33.728641 24501 tablet_service.cc:1514] Processing DeleteTablet for tablet 1edf597828e94a6787aa7aa0231080d0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 06:26:33 UTC) from {username='kudu', principal='kudu/127.21.12.190@KRBTEST.COM'} at 127.0.0.1:40218
I20260806 06:26:33.728654 24817 tablet_replica.cc:330] T 1edf597828e94a6787aa7aa0231080d0 P 0db93c757c334db8946a1e5df997c5d9: stopping tablet replica
I20260806 06:26:33.728718 24817 raft_consensus.cc:2229] T 1edf597828e94a6787aa7aa0231080d0 P 0db93c757c334db8946a1e5df997c5d9 [term 1 LEADER]: Raft consensus shutting down.
I20260806 06:26:33.728791 24817 raft_consensus.cc:2258] T 1edf597828e94a6787aa7aa0231080d0 P 0db93c757c334db8946a1e5df997c5d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 06:26:33.728832 24818 tablet_replica.cc:330] T 1edf597828e94a6787aa7aa0231080d0 P 09211067c8c14e159b2941318ccdacb5: stopping tablet replica
I20260806 06:26:33.729008 24818 raft_consensus.cc:2229] T 1edf597828e94a6787aa7aa0231080d0 P 09211067c8c14e159b2941318ccdacb5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 06:26:33.729074 24817 ts_tablet_manager.cc:1905] T 1edf597828e94a6787aa7aa0231080d0 P 0db93c757c334db8946a1e5df997c5d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 06:26:33.729091 24818 raft_consensus.cc:2258] T 1edf597828e94a6787aa7aa0231080d0 P 09211067c8c14e159b2941318ccdacb5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 06:26:33.729317 24818 ts_tablet_manager.cc:1905] T 1edf597828e94a6787aa7aa0231080d0 P 09211067c8c14e159b2941318ccdacb5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 06:26:33.730952 24817 ts_tablet_manager.cc:1918] T 1edf597828e94a6787aa7aa0231080d0 P 0db93c757c334db8946a1e5df997c5d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260806 06:26:33.731093 24817 log.cc:1196] T 1edf597828e94a6787aa7aa0231080d0 P 0db93c757c334db8946a1e5df997c5d9: Deleting WAL directory at /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785997581049705-21554-0/minicluster-data/ts-1/wal/wals/1edf597828e94a6787aa7aa0231080d0
I20260806 06:26:33.731209 24818 ts_tablet_manager.cc:1918] T 1edf597828e94a6787aa7aa0231080d0 P 09211067c8c14e159b2941318ccdacb5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260806 06:26:33.731319 24818 log.cc:1196] T 1edf597828e94a6787aa7aa0231080d0 P 09211067c8c14e159b2941318ccdacb5: Deleting WAL directory at /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785997581049705-21554-0/minicluster-data/ts-2/wal/wals/1edf597828e94a6787aa7aa0231080d0
I20260806 06:26:33.731434 24817 ts_tablet_manager.cc:1939] T 1edf597828e94a6787aa7aa0231080d0 P 0db93c757c334db8946a1e5df997c5d9: Deleting consensus metadata
I20260806 06:26:33.731639 24818 ts_tablet_manager.cc:1939] T 1edf597828e94a6787aa7aa0231080d0 P 09211067c8c14e159b2941318ccdacb5: Deleting consensus metadata
I20260806 06:26:33.731778 23862 catalog_manager.cc:4841] TS 0db93c757c334db8946a1e5df997c5d9 (127.21.12.130:43279): tablet 1edf597828e94a6787aa7aa0231080d0 (table test-table [id=ecc76554ccb440b2992fc5bdd27106cf]) successfully deleted
I20260806 06:26:33.732085 23862 catalog_manager.cc:4841] TS 09211067c8c14e159b2941318ccdacb5 (127.21.12.131:34181): tablet 1edf597828e94a6787aa7aa0231080d0 (table test-table [id=ecc76554ccb440b2992fc5bdd27106cf]) successfully deleted
W20260806 06:26:33.742501 23865 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.12.129:46063 (error 108)
I20260806 06:26:33.742761 21554 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHhLhT/build/release/bin/kudu with pid 24191
I20260806 06:26:33.747560 24820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:33.742901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.129:46063 (local address 127.0.0.1:45150)
0806 06:26:33.747460 (+ 4559us) negotiation.cc:107] Waiting for socket to connect
0806 06:26:33.747518 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.129:46063: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":4531,"thread_start_us":59,"threads_started":1}
W20260806 06:26:33.747778 23865 catalog_manager.cc:4584] TS 9447631f55b7404f9ab0232e0a1a2dc8 (127.21.12.129:46063): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1edf597828e94a6787aa7aa0231080d0: Network error: Client connection negotiation failed: client connection to 127.21.12.129:46063: connect: Connection refused (error 111)
I20260806 06:26:33.752936 21554 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHhLhT/build/release/bin/kudu with pid 24394
I20260806 06:26:33.760391 21554 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHhLhT/build/release/bin/kudu with pid 23842
2026-08-06T06:26:33Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (5427 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785997581049705-21554-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 06:26:33 dist-test-slave-4tdj krb5kdc[24824](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 06:26:33 dist-test-slave-4tdj krb5kdc[24824](info): set up 2 sockets
Aug 06 06:26:33 dist-test-slave-4tdj krb5kdc[24824](info): commencing operation
krb5kdc: starting...
W20260806 06:26:34.884356 21554 mini_kdc.cc:121] Time spent starting KDC: real 1.099s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785997581049705-21554-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785997581049705-21554-0/krb5kdc/test-user.keytab.
Aug 06 06:26:34 dist-test-slave-4tdj krb5kdc[24824](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997594, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T06:26:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T06:26:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.12.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.12.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.12.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.190.keytab.
Entry for principal kudu/127.21.12.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.190.keytab.
I20260806 06:26:35.003193 21554 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHhLhT/build/release/bin/kudu
/tmp/dist-test-taskFHhLhT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785997581049705-21554-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785997581049705-21554-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785997581049705-21554-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785997581049705-21554-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.12.190:45591
--webserver_interface=127.21.12.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.190.keytab
--principal=kudu/127.21.12.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.148:44791
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.12.190:45591
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785997581049705-21554-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785997581049705-21554-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785997581049705-21554-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 06:26:35.085423 24846 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 06:26:35.085623 24846 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 06:26:35.085654 24846 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 06:26:35.086802 24846 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 06:26:35.086843 24846 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 06:26:35.086861 24846 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 06:26:35.086877 24846 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 06:26:35.086894 24846 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 06:26:35.088376 24846 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.148:44791
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785997581049705-21554-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785997581049705-21554-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.12.190:45591
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.190.keytab
--principal=kudu/127.21.12.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.12.190:45591
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785997581049705-21554-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.12.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24846
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785997581049705-21554-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 05:11:02 UTC on ccycloud.quasar-hvzbbz.root.comops.site
build id 4569617
I20260806 06:26:35.088618 24846 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 06:26:35.088815 24846 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 06:26:35.093526 24855 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:26:35.093545 24846 server_base.cc:1034] running on GCE node
W20260806 06:26:35.093597 24853 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:26:35.093526 24857 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:26:35.093964 24846 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 06:26:35.094266 24846 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 06:26:35.095440 24846 hybrid_clock.cc:648] HybridClock initialized: now 1785997595095423 us; error 31 us; skew 500 ppm
Aug 06 06:26:35 dist-test-slave-4tdj krb5kdc[24824](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997595, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 06:26:35.097822 24846 init.cc:377] Logged in from keytab as kudu/127.21.12.190@KRBTEST.COM (short username kudu)
I20260806 06:26:35.098218 24846 webserver.cc:460] Webserver started at http://127.21.12.190:37327/ using document root <none> and password file <none>
I20260806 06:26:35.098449 24846 fs_manager.cc:359] Metadata directory not provided
I20260806 06:26:35.098506 24846 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 06:26:35.098630 24846 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 06:26:35.099623 24846 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785997581049705-21554-0/minicluster-data/master-0/data/instance:
uuid: "7fec92051c074094a780e46267e26e88"
format_stamp: "Formatted at 2026-08-06 06:26:35 on dist-test-slave-4tdj"
server_key: "e72485a92ae368ae917ea9101a2f4327"
server_key_iv: "1f6d5d33b7624488d7cd5963dba6c29e"
server_key_version: "encryptionkey@0"
I20260806 06:26:35.099951 24846 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785997581049705-21554-0/minicluster-data/master-0/wal/instance:
uuid: "7fec92051c074094a780e46267e26e88"
format_stamp: "Formatted at 2026-08-06 06:26:35 on dist-test-slave-4tdj"
server_key: "e72485a92ae368ae917ea9101a2f4327"
server_key_iv: "1f6d5d33b7624488d7cd5963dba6c29e"
server_key_version: "encryptionkey@0"
I20260806 06:26:35.101220 24846 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260806 06:26:35.101893 24867 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 06:26:35.102052 24846 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260806 06:26:35.102138 24846 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785997581049705-21554-0/minicluster-data/master-0/data,/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785997581049705-21554-0/minicluster-data/master-0/wal
uuid: "7fec92051c074094a780e46267e26e88"
format_stamp: "Formatted at 2026-08-06 06:26:35 on dist-test-slave-4tdj"
server_key: "e72485a92ae368ae917ea9101a2f4327"
server_key_iv: "1f6d5d33b7624488d7cd5963dba6c29e"
server_key_version: "encryptionkey@0"
I20260806 06:26:35.102197 24846 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785997581049705-21554-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785997581049705-21554-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785997581049705-21554-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 06:26:35.113211 24846 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 06:26:35.114979 24846 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 06:26:35.115105 24846 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 06:26:35.118820 24846 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.190:45591
I20260806 06:26:35.118853 24940 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.190:45591 every 8 connection(s)
I20260806 06:26:35.119158 24846 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785997581049705-21554-0/minicluster-data/master-0/data/info.pb
I20260806 06:26:35.120060 24942 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 06:26:35.122274 24942 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7fec92051c074094a780e46267e26e88: Bootstrap starting.
I20260806 06:26:35.122851 24942 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7fec92051c074094a780e46267e26e88: Neither blocks nor log segments found. Creating new log.
I20260806 06:26:35.123102 24942 log.cc:824] T 00000000000000000000000000000000 P 7fec92051c074094a780e46267e26e88: Log is configured to *not* fsync() on all Append() calls
I20260806 06:26:35.123770 24942 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7fec92051c074094a780e46267e26e88: No bootstrap required, opened a new log
I20260806 06:26:35.125293 24942 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7fec92051c074094a780e46267e26e88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fec92051c074094a780e46267e26e88" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 45591 } }
I20260806 06:26:35.125420 24942 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7fec92051c074094a780e46267e26e88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 06:26:35.125450 24942 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7fec92051c074094a780e46267e26e88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fec92051c074094a780e46267e26e88, State: Initialized, Role: FOLLOWER
I20260806 06:26:35.125578 24942 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7fec92051c074094a780e46267e26e88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fec92051c074094a780e46267e26e88" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 45591 } }
I20260806 06:26:35.125638 24942 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7fec92051c074094a780e46267e26e88 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 06:26:35.125669 24942 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7fec92051c074094a780e46267e26e88 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 06:26:35.125716 24942 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7fec92051c074094a780e46267e26e88 [term 0 FOLLOWER]: Advancing to term 1
I20260806 06:26:35.126199 24942 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7fec92051c074094a780e46267e26e88 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fec92051c074094a780e46267e26e88" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 45591 } }
I20260806 06:26:35.126312 24942 leader_election.cc:302] T 00000000000000000000000000000000 P 7fec92051c074094a780e46267e26e88 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7fec92051c074094a780e46267e26e88; no voters:
I20260806 06:26:35.126461 24942 leader_election.cc:288] T 00000000000000000000000000000000 P 7fec92051c074094a780e46267e26e88 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 06:26:35.126500 24945 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7fec92051c074094a780e46267e26e88 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 06:26:35.126624 24945 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7fec92051c074094a780e46267e26e88 [term 1 LEADER]: Becoming Leader. State: Replica: 7fec92051c074094a780e46267e26e88, State: Running, Role: LEADER
I20260806 06:26:35.126731 24942 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7fec92051c074094a780e46267e26e88 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 06:26:35.126760 24945 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7fec92051c074094a780e46267e26e88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fec92051c074094a780e46267e26e88" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 45591 } }
I20260806 06:26:35.127162 24946 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7fec92051c074094a780e46267e26e88 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7fec92051c074094a780e46267e26e88" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fec92051c074094a780e46267e26e88" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 45591 } } }
I20260806 06:26:35.127244 24946 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7fec92051c074094a780e46267e26e88 [sys.catalog]: This master's current role is: LEADER
I20260806 06:26:35.127282 24948 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7fec92051c074094a780e46267e26e88 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7fec92051c074094a780e46267e26e88. Latest consensus state: current_term: 1 leader_uuid: "7fec92051c074094a780e46267e26e88" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fec92051c074094a780e46267e26e88" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 45591 } } }
I20260806 06:26:35.127362 24948 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7fec92051c074094a780e46267e26e88 [sys.catalog]: This master's current role is: LEADER
I20260806 06:26:35.127688 24956 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 06:26:35.127768 21554 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHhLhT/build/release/bin/kudu as pid 24846
I20260806 06:26:35.127861 21554 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785997581049705-21554-0/minicluster-data/master-0/wal/instance
I20260806 06:26:35.128062 21554 external_mini_cluster.cc:1428] Setting key cd0eaf8300c94284bb54833a3005690d
I20260806 06:26:35.128248 24956 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 06:26:35.129768 24956 catalog_manager.cc:1347] Generated new cluster ID: e0570c0424254e0586a1172a595e700f
I20260806 06:26:35.129815 24956 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 06 06:26:35 dist-test-slave-4tdj krb5kdc[24824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997594, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.12.190@KRBTEST.COM
I20260806 06:26:35.137216 24969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:35.128750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56816 (local address 127.21.12.190:45591)
0806 06:26:35.128963 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0806 06:26:35.128973 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 06:26:35.129004 (+ 31us) server_negotiation.cc:407] Connection header received
0806 06:26:35.129122 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:26:35.129128 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:26:35.129180 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:26:35.129320 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:26:35.130014 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:35.131023 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:35.131747 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:35.132161 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:35.132253 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:35.134409 (+ 2156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:26:35.134413 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:26:35.134413 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 06:26:35.134425 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:26:35.135867 (+ 1442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:26:35.136178 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:26:35.136181 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:26:35.136182 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:26:35.136273 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:26:35.136443 (+ 170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:26:35.136443 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:26:35.136444 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:26:35.136657 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:26:35.136795 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0806 06:26:35.136890 (+ 95us) server_negotiation.cc:299] Negotiation successful
0806 06:26:35.137020 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":73,"threads_started":1}
I20260806 06:26:35.150794 24956 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 06:26:35.151311 24956 catalog_manager.cc:1504] Loading token signing keys...
I20260806 06:26:35.158658 24956 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7fec92051c074094a780e46267e26e88: Generated new TSK 0
I20260806 06:26:35.158828 24956 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.12.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.12.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.12.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.129.keytab.
Entry for principal kudu/127.21.12.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.129.keytab.
I20260806 06:26:35.189435 21554 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHhLhT/build/release/bin/kudu
/tmp/dist-test-taskFHhLhT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785997581049705-21554-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785997581049705-21554-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785997581049705-21554-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785997581049705-21554-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.12.129:0
--local_ip_for_outbound_sockets=127.21.12.129
--webserver_interface=127.21.12.129
--webserver_port=0
--tserver_master_addrs=127.21.12.190:45591
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.129.keytab
--principal=kudu/127.21.12.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.148:44791
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785997581049705-21554-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785997581049705-21554-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785997581049705-21554-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 06:26:35.270135 24973 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 06:26:35.270341 24973 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 06:26:35.270373 24973 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 06:26:35.271525 24973 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 06:26:35.271566 24973 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 06:26:35.271591 24973 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.12.129
I20260806 06:26:35.272992 24973 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.148:44791
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785997581049705-21554-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785997581049705-21554-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.129.keytab
--principal=kudu/127.21.12.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.12.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785997581049705-21554-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.12.129
--webserver_port=0
--tserver_master_addrs=127.21.12.190:45591
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24973
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.12.129
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785997581049705-21554-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 05:11:02 UTC on ccycloud.quasar-hvzbbz.root.comops.site
build id 4569617
I20260806 06:26:35.273204 24973 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 06:26:35.273407 24973 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 06:26:35.278098 24984 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:26:35.278108 24978 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:26:35.278330 24973 server_base.cc:1034] running on GCE node
W20260806 06:26:35.278337 24986 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:26:35.278573 24973 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 06:26:35.278810 24973 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 06:26:35.279947 24973 hybrid_clock.cc:648] HybridClock initialized: now 1785997595279938 us; error 31 us; skew 500 ppm
Aug 06 06:26:35 dist-test-slave-4tdj krb5kdc[24824](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997595, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 06:26:35.282392 24973 init.cc:377] Logged in from keytab as kudu/127.21.12.129@KRBTEST.COM (short username kudu)
I20260806 06:26:35.282826 24973 webserver.cc:460] Webserver started at http://127.21.12.129:37535/ using document root <none> and password file <none>
I20260806 06:26:35.283030 24973 fs_manager.cc:359] Metadata directory not provided
I20260806 06:26:35.283077 24973 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 06:26:35.283200 24973 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 06:26:35.284214 24973 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785997581049705-21554-0/minicluster-data/ts-0/data/instance:
uuid: "196e774591e94076894c8212cd0a2a07"
format_stamp: "Formatted at 2026-08-06 06:26:35 on dist-test-slave-4tdj"
server_key: "ff4a607ac0d01a29a5e7ce573dc5f50b"
server_key_iv: "4910283774f9f833d34e1772ca46e1de"
server_key_version: "encryptionkey@0"
I20260806 06:26:35.284544 24973 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785997581049705-21554-0/minicluster-data/ts-0/wal/instance:
uuid: "196e774591e94076894c8212cd0a2a07"
format_stamp: "Formatted at 2026-08-06 06:26:35 on dist-test-slave-4tdj"
server_key: "ff4a607ac0d01a29a5e7ce573dc5f50b"
server_key_iv: "4910283774f9f833d34e1772ca46e1de"
server_key_version: "encryptionkey@0"
I20260806 06:26:35.285883 24973 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260806 06:26:35.286675 24995 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 06:26:35.286844 24973 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260806 06:26:35.286931 24973 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785997581049705-21554-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785997581049705-21554-0/minicluster-data/ts-0/wal
uuid: "196e774591e94076894c8212cd0a2a07"
format_stamp: "Formatted at 2026-08-06 06:26:35 on dist-test-slave-4tdj"
server_key: "ff4a607ac0d01a29a5e7ce573dc5f50b"
server_key_iv: "4910283774f9f833d34e1772ca46e1de"
server_key_version: "encryptionkey@0"
I20260806 06:26:35.286988 24973 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785997581049705-21554-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785997581049705-21554-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785997581049705-21554-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 06:26:35.295408 24973 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 06:26:35.297399 24973 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 06:26:35.297536 24973 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 06:26:35.297739 24973 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 06:26:35.298017 24973 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 06:26:35.298059 24973 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 06:26:35.298085 24973 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 06:26:35.298102 24973 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 06:26:35.303725 24973 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.129:36655
I20260806 06:26:35.303771 25141 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.129:36655 every 8 connection(s)
I20260806 06:26:35.304116 24973 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785997581049705-21554-0/minicluster-data/ts-0/data/info.pb
Aug 06 06:26:35 dist-test-slave-4tdj krb5kdc[24824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997595, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.129@KRBTEST.COM for kudu/127.21.12.190@KRBTEST.COM
I20260806 06:26:35.313131 24969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:35.304825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.129:36517 (local address 127.21.12.190:45591)
0806 06:26:35.304901 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0806 06:26:35.304904 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 06:26:35.305116 (+ 212us) server_negotiation.cc:407] Connection header received
0806 06:26:35.305692 (+ 576us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:26:35.305693 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:26:35.305708 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:26:35.305783 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:26:35.306791 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:35.307307 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:35.308286 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:35.308519 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:35.308590 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:35.310633 (+ 2043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:26:35.310634 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:26:35.310635 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 06:26:35.310644 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:26:35.311713 (+ 1069us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:26:35.312143 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:26:35.312144 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:26:35.312145 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:26:35.312208 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:26:35.312502 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:26:35.312503 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:26:35.312504 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:26:35.312628 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:26:35.312694 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0806 06:26:35.312967 (+ 273us) server_negotiation.cc:299] Negotiation successful
0806 06:26:35.313065 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260806 06:26:35.313184 25146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:35.304822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:45591 (local address 127.21.12.129:36517)
0806 06:26:35.304994 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0806 06:26:35.305037 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0806 06:26:35.305589 (+ 552us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:26:35.305865 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:26:35.305870 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:26:35.306490 (+ 620us) client_negotiation.cc:189] Negotiated authn=SASL
0806 06:26:35.306688 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:35.306693 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:35.307376 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:35.307376 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:35.308190 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:35.308192 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:35.308606 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:35.308607 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:35.308708 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:35.309144 (+ 436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 06:26:35.309152 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0806 06:26:35.310537 (+ 1385us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 06:26:35.311807 (+ 1270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:26:35.311815 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:26:35.311816 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:26:35.312057 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:26:35.312296 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:26:35.312297 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:26:35.312297 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:26:35.312409 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:26:35.312697 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 06:26:35.312703 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 06:26:35.312818 (+ 115us) client_negotiation.cc:769] Sending connection context
0806 06:26:35.312902 (+ 84us) client_negotiation.cc:240] Negotiation successful
0806 06:26:35.313025 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":89,"threads_started":1}
I20260806 06:26:35.313691 25142 heartbeater.cc:347] Connected to a master server at 127.21.12.190:45591
I20260806 06:26:35.313788 25142 heartbeater.cc:464] Registering TS with master...
I20260806 06:26:35.313966 25142 heartbeater.cc:511] Master 127.21.12.190:45591 requested a full tablet report, sending...
I20260806 06:26:35.314168 21554 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHhLhT/build/release/bin/kudu as pid 24973
I20260806 06:26:35.314262 21554 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785997581049705-21554-0/minicluster-data/ts-0/wal/instance
I20260806 06:26:35.314465 21554 external_mini_cluster.cc:1428] Setting key d5604a50eafa30038fcde47d17efdf21
I20260806 06:26:35.314587 24892 ts_manager.cc:194] Registered new tserver with Master: 196e774591e94076894c8212cd0a2a07 (127.21.12.129:36655)
I20260806 06:26:35.315362 24892 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.12.129@KRBTEST.COM'} at 127.21.12.129:36517
WARNING: no policy specified for kudu/127.21.12.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.12.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.12.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.130.keytab.
Entry for principal kudu/127.21.12.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.130.keytab.
I20260806 06:26:35.339141 21554 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHhLhT/build/release/bin/kudu
/tmp/dist-test-taskFHhLhT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785997581049705-21554-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785997581049705-21554-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785997581049705-21554-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785997581049705-21554-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.12.130:0
--local_ip_for_outbound_sockets=127.21.12.130
--webserver_interface=127.21.12.130
--webserver_port=0
--tserver_master_addrs=127.21.12.190:45591
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.130.keytab
--principal=kudu/127.21.12.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.148:44791
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785997581049705-21554-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785997581049705-21554-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785997581049705-21554-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 06:26:35.421875 25150 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 06:26:35.422091 25150 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 06:26:35.422115 25150 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 06:26:35.423306 25150 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 06:26:35.423358 25150 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 06:26:35.423385 25150 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.12.130
I20260806 06:26:35.424826 25150 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.148:44791
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785997581049705-21554-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785997581049705-21554-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.130.keytab
--principal=kudu/127.21.12.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.12.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785997581049705-21554-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.12.130
--webserver_port=0
--tserver_master_addrs=127.21.12.190:45591
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25150
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.12.130
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785997581049705-21554-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 05:11:02 UTC on ccycloud.quasar-hvzbbz.root.comops.site
build id 4569617
I20260806 06:26:35.425051 25150 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 06:26:35.425280 25150 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 06:26:35.430438 25159 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:26:35.430410 25157 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:26:35.430469 25150 server_base.cc:1034] running on GCE node
W20260806 06:26:35.430440 25156 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:26:35.430850 25150 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 06:26:35.431092 25150 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 06:26:35.432221 25150 hybrid_clock.cc:648] HybridClock initialized: now 1785997595432214 us; error 30 us; skew 500 ppm
Aug 06 06:26:35 dist-test-slave-4tdj krb5kdc[24824](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997595, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 06:26:35.434686 25150 init.cc:377] Logged in from keytab as kudu/127.21.12.130@KRBTEST.COM (short username kudu)
I20260806 06:26:35.435068 25150 webserver.cc:460] Webserver started at http://127.21.12.130:40505/ using document root <none> and password file <none>
I20260806 06:26:35.435287 25150 fs_manager.cc:359] Metadata directory not provided
I20260806 06:26:35.435333 25150 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 06:26:35.435451 25150 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 06:26:35.436434 25150 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785997581049705-21554-0/minicluster-data/ts-1/data/instance:
uuid: "e71dc09b5f1049f299a266f21c405831"
format_stamp: "Formatted at 2026-08-06 06:26:35 on dist-test-slave-4tdj"
server_key: "94e11c585fed50c43a1d720b55b99862"
server_key_iv: "4317b79c91c400f18019e9e8d6d23cd7"
server_key_version: "encryptionkey@0"
I20260806 06:26:35.436769 25150 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785997581049705-21554-0/minicluster-data/ts-1/wal/instance:
uuid: "e71dc09b5f1049f299a266f21c405831"
format_stamp: "Formatted at 2026-08-06 06:26:35 on dist-test-slave-4tdj"
server_key: "94e11c585fed50c43a1d720b55b99862"
server_key_iv: "4317b79c91c400f18019e9e8d6d23cd7"
server_key_version: "encryptionkey@0"
I20260806 06:26:35.438042 25150 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260806 06:26:35.438846 25169 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 06:26:35.439004 25150 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260806 06:26:35.439080 25150 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785997581049705-21554-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785997581049705-21554-0/minicluster-data/ts-1/wal
uuid: "e71dc09b5f1049f299a266f21c405831"
format_stamp: "Formatted at 2026-08-06 06:26:35 on dist-test-slave-4tdj"
server_key: "94e11c585fed50c43a1d720b55b99862"
server_key_iv: "4317b79c91c400f18019e9e8d6d23cd7"
server_key_version: "encryptionkey@0"
I20260806 06:26:35.439148 25150 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785997581049705-21554-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785997581049705-21554-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785997581049705-21554-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 06:26:35.447917 25150 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 06:26:35.449725 25150 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 06:26:35.449855 25150 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 06:26:35.450069 25150 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 06:26:35.450428 25150 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 06:26:35.450466 25150 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 06:26:35.450506 25150 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 06:26:35.450528 25150 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 06:26:35.456104 25150 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.130:32947
I20260806 06:26:35.456161 25313 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.130:32947 every 8 connection(s)
I20260806 06:26:35.456454 25150 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785997581049705-21554-0/minicluster-data/ts-1/data/info.pb
Aug 06 06:26:35 dist-test-slave-4tdj krb5kdc[24824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997595, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.130@KRBTEST.COM for kudu/127.21.12.190@KRBTEST.COM
I20260806 06:26:35.463800 21554 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHhLhT/build/release/bin/kudu as pid 25150
I20260806 06:26:35.463892 21554 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785997581049705-21554-0/minicluster-data/ts-1/wal/instance
I20260806 06:26:35.464059 21554 external_mini_cluster.cc:1428] Setting key becb367275c77aee103758217f93b248
I20260806 06:26:35.465256 24969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:35.457204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.130:35345 (local address 127.21.12.190:45591)
0806 06:26:35.457261 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0806 06:26:35.457262 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0806 06:26:35.457450 (+ 188us) server_negotiation.cc:407] Connection header received
0806 06:26:35.457904 (+ 454us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:26:35.457905 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:26:35.457914 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:26:35.457967 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:26:35.458901 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:35.459404 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:35.460337 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:35.460544 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:35.460599 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:35.462535 (+ 1936us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:26:35.462537 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:26:35.462538 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 06:26:35.462544 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:26:35.463844 (+ 1300us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:26:35.464212 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:26:35.464213 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:26:35.464214 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:26:35.464268 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:26:35.464539 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:26:35.464539 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:26:35.464540 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:26:35.464649 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:26:35.464711 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0806 06:26:35.465110 (+ 399us) server_negotiation.cc:299] Negotiation successful
0806 06:26:35.465178 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260806 06:26:35.465474 25317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:35.457167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:45591 (local address 127.21.12.130:35345)
0806 06:26:35.457330 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0806 06:26:35.457373 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0806 06:26:35.457831 (+ 458us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:26:35.458033 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:26:35.458036 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:26:35.458627 (+ 591us) client_negotiation.cc:189] Negotiated authn=SASL
0806 06:26:35.458814 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:35.458820 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:35.459476 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:35.459477 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:35.460260 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:35.460262 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:35.460603 (+ 341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:35.460604 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:35.460695 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:35.461112 (+ 417us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 06:26:35.461120 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0806 06:26:35.462449 (+ 1329us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 06:26:35.463923 (+ 1474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:26:35.463930 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:26:35.463932 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:26:35.464145 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:26:35.464335 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:26:35.464336 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:26:35.464336 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:26:35.464474 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:26:35.464803 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 06:26:35.464810 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 06:26:35.464967 (+ 157us) client_negotiation.cc:769] Sending connection context
0806 06:26:35.465098 (+ 131us) client_negotiation.cc:240] Negotiation successful
0806 06:26:35.465262 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":78,"threads_started":1}
I20260806 06:26:35.465898 25315 heartbeater.cc:347] Connected to a master server at 127.21.12.190:45591
I20260806 06:26:35.465993 25315 heartbeater.cc:464] Registering TS with master...
I20260806 06:26:35.466158 25315 heartbeater.cc:511] Master 127.21.12.190:45591 requested a full tablet report, sending...
I20260806 06:26:35.466548 24892 ts_manager.cc:194] Registered new tserver with Master: e71dc09b5f1049f299a266f21c405831 (127.21.12.130:32947)
I20260806 06:26:35.467026 24892 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.12.130@KRBTEST.COM'} at 127.21.12.130:35345
WARNING: no policy specified for kudu/127.21.12.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.12.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.12.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.131.keytab.
Entry for principal kudu/127.21.12.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.131.keytab.
I20260806 06:26:35.489454 21554 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHhLhT/build/release/bin/kudu
/tmp/dist-test-taskFHhLhT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785997581049705-21554-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785997581049705-21554-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785997581049705-21554-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785997581049705-21554-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.12.131:0
--local_ip_for_outbound_sockets=127.21.12.131
--webserver_interface=127.21.12.131
--webserver_port=0
--tserver_master_addrs=127.21.12.190:45591
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.131.keytab
--principal=kudu/127.21.12.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.148:44791
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785997581049705-21554-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785997581049705-21554-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785997581049705-21554-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 06:26:35.568419 25322 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 06:26:35.568612 25322 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 06:26:35.568634 25322 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 06:26:35.569710 25322 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 06:26:35.569739 25322 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 06:26:35.569762 25322 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.12.131
I20260806 06:26:35.571126 25322 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.148:44791
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785997581049705-21554-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785997581049705-21554-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.131.keytab
--principal=kudu/127.21.12.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.12.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785997581049705-21554-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.12.131
--webserver_port=0
--tserver_master_addrs=127.21.12.190:45591
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25322
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.12.131
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785997581049705-21554-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 05:11:02 UTC on ccycloud.quasar-hvzbbz.root.comops.site
build id 4569617
I20260806 06:26:35.571310 25322 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 06:26:35.571475 25322 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 06:26:35.576184 25332 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:26:35.576207 25329 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:26:35.576299 25328 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:26:35.576345 25322 server_base.cc:1034] running on GCE node
I20260806 06:26:35.576625 25322 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 06:26:35.576836 25322 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 06:26:35.577975 25322 hybrid_clock.cc:648] HybridClock initialized: now 1785997595577958 us; error 28 us; skew 500 ppm
Aug 06 06:26:35 dist-test-slave-4tdj krb5kdc[24824](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997595, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 06:26:35.580283 25322 init.cc:377] Logged in from keytab as kudu/127.21.12.131@KRBTEST.COM (short username kudu)
I20260806 06:26:35.580711 25322 webserver.cc:460] Webserver started at http://127.21.12.131:34511/ using document root <none> and password file <none>
I20260806 06:26:35.580901 25322 fs_manager.cc:359] Metadata directory not provided
I20260806 06:26:35.580947 25322 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 06:26:35.581071 25322 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 06:26:35.582024 25322 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785997581049705-21554-0/minicluster-data/ts-2/data/instance:
uuid: "83562200fa4d4e918a2133d334c41064"
format_stamp: "Formatted at 2026-08-06 06:26:35 on dist-test-slave-4tdj"
server_key: "6859ff4cd80223719f9fb4c370460532"
server_key_iv: "cea7626e874327623cdbafa73c1871a7"
server_key_version: "encryptionkey@0"
I20260806 06:26:35.582381 25322 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785997581049705-21554-0/minicluster-data/ts-2/wal/instance:
uuid: "83562200fa4d4e918a2133d334c41064"
format_stamp: "Formatted at 2026-08-06 06:26:35 on dist-test-slave-4tdj"
server_key: "6859ff4cd80223719f9fb4c370460532"
server_key_iv: "cea7626e874327623cdbafa73c1871a7"
server_key_version: "encryptionkey@0"
I20260806 06:26:35.583601 25322 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260806 06:26:35.584363 25340 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 06:26:35.584529 25322 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 06:26:35.584622 25322 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785997581049705-21554-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785997581049705-21554-0/minicluster-data/ts-2/wal
uuid: "83562200fa4d4e918a2133d334c41064"
format_stamp: "Formatted at 2026-08-06 06:26:35 on dist-test-slave-4tdj"
server_key: "6859ff4cd80223719f9fb4c370460532"
server_key_iv: "cea7626e874327623cdbafa73c1871a7"
server_key_version: "encryptionkey@0"
I20260806 06:26:35.584692 25322 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785997581049705-21554-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785997581049705-21554-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785997581049705-21554-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 06:26:35.597470 25322 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 06:26:35.599126 25322 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 06:26:35.599234 25322 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 06:26:35.599440 25322 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 06:26:35.599717 25322 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 06:26:35.599750 25322 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 06:26:35.599771 25322 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 06:26:35.599793 25322 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 06:26:35.605315 25322 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.131:44767
I20260806 06:26:35.605372 25511 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.131:44767 every 8 connection(s)
I20260806 06:26:35.605691 25322 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785997581049705-21554-0/minicluster-data/ts-2/data/info.pb
Aug 06 06:26:35 dist-test-slave-4tdj krb5kdc[24824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997595, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.131@KRBTEST.COM for kudu/127.21.12.190@KRBTEST.COM
I20260806 06:26:35.613871 24969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:35.606399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.131:32947 (local address 127.21.12.190:45591)
0806 06:26:35.606462 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0806 06:26:35.606465 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 06:26:35.606688 (+ 223us) server_negotiation.cc:407] Connection header received
0806 06:26:35.607138 (+ 450us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:26:35.607139 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:26:35.607149 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:26:35.607197 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:26:35.608054 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:35.608548 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:35.609420 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:35.609614 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:35.609662 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:35.611507 (+ 1845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:26:35.611509 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:26:35.611509 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 06:26:35.611516 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:26:35.612745 (+ 1229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:26:35.613073 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:26:35.613074 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:26:35.613074 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:26:35.613118 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:26:35.613283 (+ 165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:26:35.613283 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:26:35.613284 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:26:35.613372 (+ 88us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:26:35.613416 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0806 06:26:35.613755 (+ 339us) server_negotiation.cc:299] Negotiation successful
0806 06:26:35.613810 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260806 06:26:35.614048 25516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:35.606416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:45591 (local address 127.21.12.131:32947)
0806 06:26:35.606571 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0806 06:26:35.606613 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0806 06:26:35.607056 (+ 443us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:26:35.607253 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:26:35.607256 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:26:35.607805 (+ 549us) client_negotiation.cc:189] Negotiated authn=SASL
0806 06:26:35.607982 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:35.607987 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:35.608615 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:35.608616 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:35.609364 (+ 748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:35.609366 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:35.609672 (+ 306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:35.609672 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:35.609761 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:35.610162 (+ 401us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 06:26:35.610170 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0806 06:26:35.611429 (+ 1259us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 06:26:35.612822 (+ 1393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:26:35.612828 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:26:35.612829 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:26:35.613020 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:26:35.613158 (+ 138us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:26:35.613159 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:26:35.613159 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:26:35.613244 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:26:35.613500 (+ 256us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 06:26:35.613507 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 06:26:35.613621 (+ 114us) client_negotiation.cc:769] Sending connection context
0806 06:26:35.613728 (+ 107us) client_negotiation.cc:240] Negotiation successful
0806 06:26:35.613843 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":87,"threads_started":1}
I20260806 06:26:35.614450 21554 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHhLhT/build/release/bin/kudu as pid 25322
I20260806 06:26:35.614550 25513 heartbeater.cc:347] Connected to a master server at 127.21.12.190:45591
I20260806 06:26:35.614573 21554 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785997581049705-21554-0/minicluster-data/ts-2/wal/instance
I20260806 06:26:35.614638 25513 heartbeater.cc:464] Registering TS with master...
I20260806 06:26:35.614776 21554 external_mini_cluster.cc:1428] Setting key 4273d566f228095bb5b59ee95a6c2f18
I20260806 06:26:35.614802 25513 heartbeater.cc:511] Master 127.21.12.190:45591 requested a full tablet report, sending...
I20260806 06:26:35.615182 24891 ts_manager.cc:194] Registered new tserver with Master: 83562200fa4d4e918a2133d334c41064 (127.21.12.131:44767)
I20260806 06:26:35.615619 21554 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 06:26:35.615638 24891 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.12.131@KRBTEST.COM'} at 127.21.12.131:32947
W20260806 06:26:35.628006 24969 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 06:26:35.627106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56820 (local address 127.21.12.190:45591)
0806 06:26:35.627173 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0806 06:26:35.627175 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:26:35.627266 (+ 91us) server_negotiation.cc:407] Connection header received
0806 06:26:35.627332 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:26:35.627333 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:26:35.627341 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:26:35.627387 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:26:35.627916 (+ 529us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56820: BlockingRecv error: recv got EOF from 127.0.0.1:56820 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
I20260806 06:26:35.628909 21554 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHhLhT/build/release/bin/kudu with pid 24973
I20260806 06:26:35.647658 21554 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHhLhT/build/release/bin/kudu with pid 25150
I20260806 06:26:35.653899 21554 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHhLhT/build/release/bin/kudu with pid 25322
I20260806 06:26:35.664129 21554 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHhLhT/build/release/bin/kudu with pid 24846
2026-08-06T06:26:35Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (1903 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785997581049705-21554-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 06:26:35 dist-test-slave-4tdj krb5kdc[25529](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 06:26:35 dist-test-slave-4tdj krb5kdc[25529](info): set up 2 sockets
Aug 06 06:26:35 dist-test-slave-4tdj krb5kdc[25529](info): commencing operation
krb5kdc: starting...
W20260806 06:26:36.778439 21554 mini_kdc.cc:121] Time spent starting KDC: real 1.090s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785997581049705-21554-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785997581049705-21554-0/krb5kdc/test-user.keytab.
Aug 06 06:26:36 dist-test-slave-4tdj krb5kdc[25529](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997596, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T06:26:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T06:26:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.12.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.12.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.12.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.190.keytab.
Entry for principal kudu/127.21.12.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.190.keytab.
I20260806 06:26:36.893975 21554 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHhLhT/build/release/bin/kudu
/tmp/dist-test-taskFHhLhT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785997581049705-21554-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785997581049705-21554-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785997581049705-21554-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785997581049705-21554-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.12.190:35893
--webserver_interface=127.21.12.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.190.keytab
--principal=kudu/127.21.12.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.148:37765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.12.190:35893
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785997581049705-21554-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785997581049705-21554-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785997581049705-21554-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 06:26:36.972677 25549 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 06:26:36.972870 25549 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 06:26:36.972899 25549 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 06:26:36.973999 25549 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 06:26:36.974037 25549 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 06:26:36.974056 25549 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 06:26:36.974071 25549 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 06:26:36.974086 25549 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 06:26:36.975556 25549 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.148:37765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785997581049705-21554-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785997581049705-21554-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.12.190:35893
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.190.keytab
--principal=kudu/127.21.12.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.12.190:35893
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785997581049705-21554-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.12.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25549
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785997581049705-21554-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 05:11:02 UTC on ccycloud.quasar-hvzbbz.root.comops.site
build id 4569617
I20260806 06:26:36.975780 25549 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 06:26:36.975966 25549 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 06:26:36.980626 25558 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:26:36.980597 25560 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:26:36.980639 25557 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:26:36.980759 25549 server_base.cc:1034] running on GCE node
I20260806 06:26:36.981079 25549 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 06:26:36.981333 25549 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 06:26:36.982487 25549 hybrid_clock.cc:648] HybridClock initialized: now 1785997596982466 us; error 34 us; skew 500 ppm
Aug 06 06:26:36 dist-test-slave-4tdj krb5kdc[25529](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997596, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 06:26:36.984903 25549 init.cc:377] Logged in from keytab as kudu/127.21.12.190@KRBTEST.COM (short username kudu)
I20260806 06:26:36.985355 25549 webserver.cc:460] Webserver started at http://127.21.12.190:39639/ using document root <none> and password file <none>
I20260806 06:26:36.985538 25549 fs_manager.cc:359] Metadata directory not provided
I20260806 06:26:36.985579 25549 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 06:26:36.985678 25549 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 06:26:36.986701 25549 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785997581049705-21554-0/minicluster-data/master-0/data/instance:
uuid: "7c7d3ea447f64e8f943fae2340d21afe"
format_stamp: "Formatted at 2026-08-06 06:26:36 on dist-test-slave-4tdj"
server_key: "1e5b83a7f10862968a0411803265bf6f"
server_key_iv: "d0c1bd3a2f640e0e0bf62ad86371eecf"
server_key_version: "encryptionkey@0"
I20260806 06:26:36.987026 25549 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785997581049705-21554-0/minicluster-data/master-0/wal/instance:
uuid: "7c7d3ea447f64e8f943fae2340d21afe"
format_stamp: "Formatted at 2026-08-06 06:26:36 on dist-test-slave-4tdj"
server_key: "1e5b83a7f10862968a0411803265bf6f"
server_key_iv: "d0c1bd3a2f640e0e0bf62ad86371eecf"
server_key_version: "encryptionkey@0"
I20260806 06:26:36.988274 25549 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260806 06:26:36.988955 25569 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 06:26:36.989087 25549 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260806 06:26:36.989179 25549 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785997581049705-21554-0/minicluster-data/master-0/data,/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785997581049705-21554-0/minicluster-data/master-0/wal
uuid: "7c7d3ea447f64e8f943fae2340d21afe"
format_stamp: "Formatted at 2026-08-06 06:26:36 on dist-test-slave-4tdj"
server_key: "1e5b83a7f10862968a0411803265bf6f"
server_key_iv: "d0c1bd3a2f640e0e0bf62ad86371eecf"
server_key_version: "encryptionkey@0"
I20260806 06:26:36.989245 25549 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785997581049705-21554-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785997581049705-21554-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785997581049705-21554-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 06:26:37.001314 25549 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 06:26:37.003180 25549 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 06:26:37.003310 25549 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 06:26:37.007418 25549 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.190:35893
I20260806 06:26:37.007447 25632 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.190:35893 every 8 connection(s)
I20260806 06:26:37.007980 25549 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785997581049705-21554-0/minicluster-data/master-0/data/info.pb
I20260806 06:26:37.008616 25633 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 06:26:37.010946 25633 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c7d3ea447f64e8f943fae2340d21afe: Bootstrap starting.
I20260806 06:26:37.011526 25633 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7c7d3ea447f64e8f943fae2340d21afe: Neither blocks nor log segments found. Creating new log.
I20260806 06:26:37.011796 25633 log.cc:824] T 00000000000000000000000000000000 P 7c7d3ea447f64e8f943fae2340d21afe: Log is configured to *not* fsync() on all Append() calls
I20260806 06:26:37.012440 25633 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c7d3ea447f64e8f943fae2340d21afe: No bootstrap required, opened a new log
I20260806 06:26:37.013940 25633 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7c7d3ea447f64e8f943fae2340d21afe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c7d3ea447f64e8f943fae2340d21afe" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 35893 } }
I20260806 06:26:37.014089 25633 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7c7d3ea447f64e8f943fae2340d21afe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 06:26:37.014111 25633 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7c7d3ea447f64e8f943fae2340d21afe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c7d3ea447f64e8f943fae2340d21afe, State: Initialized, Role: FOLLOWER
I20260806 06:26:37.014242 25633 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7c7d3ea447f64e8f943fae2340d21afe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c7d3ea447f64e8f943fae2340d21afe" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 35893 } }
I20260806 06:26:37.014348 25633 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7c7d3ea447f64e8f943fae2340d21afe [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 06:26:37.014374 25633 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7c7d3ea447f64e8f943fae2340d21afe [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 06:26:37.014407 25633 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7c7d3ea447f64e8f943fae2340d21afe [term 0 FOLLOWER]: Advancing to term 1
I20260806 06:26:37.015264 25633 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7c7d3ea447f64e8f943fae2340d21afe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c7d3ea447f64e8f943fae2340d21afe" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 35893 } }
I20260806 06:26:37.015352 25633 leader_election.cc:302] T 00000000000000000000000000000000 P 7c7d3ea447f64e8f943fae2340d21afe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7c7d3ea447f64e8f943fae2340d21afe; no voters:
I20260806 06:26:37.015501 25633 leader_election.cc:288] T 00000000000000000000000000000000 P 7c7d3ea447f64e8f943fae2340d21afe [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 06:26:37.015592 25639 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7c7d3ea447f64e8f943fae2340d21afe [term 1 FOLLOWER]: Leader election won for term 1
I20260806 06:26:37.015744 25633 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7c7d3ea447f64e8f943fae2340d21afe [sys.catalog]: configured and running, proceeding with master startup.
I20260806 06:26:37.015776 25639 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7c7d3ea447f64e8f943fae2340d21afe [term 1 LEADER]: Becoming Leader. State: Replica: 7c7d3ea447f64e8f943fae2340d21afe, State: Running, Role: LEADER
I20260806 06:26:37.015928 25639 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7c7d3ea447f64e8f943fae2340d21afe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c7d3ea447f64e8f943fae2340d21afe" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 35893 } }
I20260806 06:26:37.016292 25641 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c7d3ea447f64e8f943fae2340d21afe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7c7d3ea447f64e8f943fae2340d21afe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c7d3ea447f64e8f943fae2340d21afe" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 35893 } } }
I20260806 06:26:37.016374 25641 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c7d3ea447f64e8f943fae2340d21afe [sys.catalog]: This master's current role is: LEADER
I20260806 06:26:37.016605 25643 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c7d3ea447f64e8f943fae2340d21afe [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7c7d3ea447f64e8f943fae2340d21afe. Latest consensus state: current_term: 1 leader_uuid: "7c7d3ea447f64e8f943fae2340d21afe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c7d3ea447f64e8f943fae2340d21afe" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 35893 } } }
I20260806 06:26:37.016696 25643 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c7d3ea447f64e8f943fae2340d21afe [sys.catalog]: This master's current role is: LEADER
I20260806 06:26:37.016894 25653 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 06:26:37.017376 25653 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 06:26:37.018143 21554 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHhLhT/build/release/bin/kudu as pid 25549
I20260806 06:26:37.018224 21554 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785997581049705-21554-0/minicluster-data/master-0/wal/instance
I20260806 06:26:37.018411 21554 external_mini_cluster.cc:1428] Setting key 3471a98ddb2248bca02e3baa184f9545
I20260806 06:26:37.018834 25653 catalog_manager.cc:1347] Generated new cluster ID: a41e5f14967146c7b2335cbae11a91f5
I20260806 06:26:37.018880 25653 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 06 06:26:37 dist-test-slave-4tdj krb5kdc[25529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997596, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.12.190@KRBTEST.COM
I20260806 06:26:37.030938 25668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:37.019080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53028 (local address 127.21.12.190:35893)
0806 06:26:37.019241 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0806 06:26:37.019250 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 06:26:37.019341 (+ 91us) server_negotiation.cc:407] Connection header received
0806 06:26:37.019536 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:26:37.019553 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:26:37.019614 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:26:37.019768 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:26:37.020589 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:37.022073 (+ 1484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:37.023305 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:37.023930 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:37.024122 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:37.027313 (+ 3191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:26:37.027317 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:26:37.027318 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 06:26:37.027330 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:26:37.029245 (+ 1915us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:26:37.029633 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:26:37.029635 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:26:37.029636 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:26:37.029740 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:26:37.029910 (+ 170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:26:37.029911 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:26:37.029911 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:26:37.030206 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:26:37.030389 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0806 06:26:37.030458 (+ 69us) server_negotiation.cc:299] Negotiation successful
0806 06:26:37.030702 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":78,"threads_started":1}
I20260806 06:26:37.031806 21554 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 06:26:37.035288 25653 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 06:26:37.035792 25653 catalog_manager.cc:1504] Loading token signing keys...
I20260806 06:26:37.048372 25668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:37.044404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53044 (local address 127.21.12.190:35893)
0806 06:26:37.044439 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0806 06:26:37.044440 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0806 06:26:37.044581 (+ 141us) server_negotiation.cc:407] Connection header received
0806 06:26:37.044615 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:26:37.044617 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:26:37.044628 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:26:37.044686 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:26:37.045062 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:37.045850 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:37.046867 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:37.047201 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:37.047270 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:37.047919 (+ 649us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:26:37.047920 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:26:37.047921 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 06:26:37.048138 (+ 217us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:26:37.048156 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 06:26:37.048164 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:26:37.048223 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0806 06:26:37.048264 (+ 41us) server_negotiation.cc:299] Negotiation successful
0806 06:26:37.048292 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260806 06:26:37.052650 25653 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7c7d3ea447f64e8f943fae2340d21afe: Generated new TSK 0
I20260806 06:26:37.052824 25653 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 06:26:37.055449 21554 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHhLhT/build/release/bin/kudu with pid 25549
2026-08-06T06:26:37Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1390 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785997581049705-21554-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 06:26:37 dist-test-slave-4tdj krb5kdc[25683](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 06:26:37 dist-test-slave-4tdj krb5kdc[25683](info): set up 2 sockets
Aug 06 06:26:37 dist-test-slave-4tdj krb5kdc[25683](info): commencing operation
krb5kdc: starting...
W20260806 06:26:38.169384 21554 mini_kdc.cc:121] Time spent starting KDC: real 1.091s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785997581049705-21554-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785997581049705-21554-0/krb5kdc/test-user.keytab.
Aug 06 06:26:38 dist-test-slave-4tdj krb5kdc[25683](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997598, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T06:26:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T06:26:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.12.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.12.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.12.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.190.keytab.
Entry for principal kudu/127.21.12.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.190.keytab.
I20260806 06:26:38.282869 21554 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHhLhT/build/release/bin/kudu
/tmp/dist-test-taskFHhLhT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785997581049705-21554-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785997581049705-21554-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785997581049705-21554-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785997581049705-21554-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.12.190:41799
--webserver_interface=127.21.12.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.190.keytab
--principal=kudu/127.21.12.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.148:35701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.12.190:41799
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785997581049705-21554-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785997581049705-21554-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785997581049705-21554-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 06:26:38.362212 25699 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 06:26:38.362422 25699 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 06:26:38.362449 25699 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 06:26:38.363557 25699 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 06:26:38.363597 25699 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 06:26:38.363615 25699 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 06:26:38.363631 25699 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 06:26:38.363644 25699 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 06:26:38.365088 25699 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.148:35701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785997581049705-21554-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785997581049705-21554-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.12.190:41799
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.190.keytab
--principal=kudu/127.21.12.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.12.190:41799
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785997581049705-21554-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.12.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25699
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785997581049705-21554-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 05:11:02 UTC on ccycloud.quasar-hvzbbz.root.comops.site
build id 4569617
I20260806 06:26:38.365334 25699 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 06:26:38.365542 25699 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260806 06:26:38.370136 25699 server_base.cc:1034] running on GCE node
W20260806 06:26:38.370119 25706 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:26:38.370097 25705 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:26:38.370087 25708 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:26:38.370541 25699 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 06:26:38.370827 25699 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 06:26:38.371963 25699 hybrid_clock.cc:648] HybridClock initialized: now 1785997598371950 us; error 28 us; skew 500 ppm
Aug 06 06:26:38 dist-test-slave-4tdj krb5kdc[25683](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997598, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 06:26:38.374279 25699 init.cc:377] Logged in from keytab as kudu/127.21.12.190@KRBTEST.COM (short username kudu)
I20260806 06:26:38.374714 25699 webserver.cc:460] Webserver started at http://127.21.12.190:42033/ using document root <none> and password file <none>
I20260806 06:26:38.374928 25699 fs_manager.cc:359] Metadata directory not provided
I20260806 06:26:38.374974 25699 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 06:26:38.375108 25699 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 06:26:38.376085 25699 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785997581049705-21554-0/minicluster-data/master-0/data/instance:
uuid: "95aa6306150c4ed087da78c36c9bd59f"
format_stamp: "Formatted at 2026-08-06 06:26:38 on dist-test-slave-4tdj"
server_key: "ac7fd4f547828d31e2b50097297f1eff"
server_key_iv: "dc451f350d818fc68df29c7824d06115"
server_key_version: "encryptionkey@0"
I20260806 06:26:38.376401 25699 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785997581049705-21554-0/minicluster-data/master-0/wal/instance:
uuid: "95aa6306150c4ed087da78c36c9bd59f"
format_stamp: "Formatted at 2026-08-06 06:26:38 on dist-test-slave-4tdj"
server_key: "ac7fd4f547828d31e2b50097297f1eff"
server_key_iv: "dc451f350d818fc68df29c7824d06115"
server_key_version: "encryptionkey@0"
I20260806 06:26:38.377588 25699 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260806 06:26:38.378203 25722 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 06:26:38.378381 25699 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260806 06:26:38.378466 25699 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785997581049705-21554-0/minicluster-data/master-0/data,/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785997581049705-21554-0/minicluster-data/master-0/wal
uuid: "95aa6306150c4ed087da78c36c9bd59f"
format_stamp: "Formatted at 2026-08-06 06:26:38 on dist-test-slave-4tdj"
server_key: "ac7fd4f547828d31e2b50097297f1eff"
server_key_iv: "dc451f350d818fc68df29c7824d06115"
server_key_version: "encryptionkey@0"
I20260806 06:26:38.378530 25699 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785997581049705-21554-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785997581049705-21554-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785997581049705-21554-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 06:26:38.391302 25699 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 06:26:38.393301 25699 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 06:26:38.393432 25699 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 06:26:38.397219 25699 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.190:41799
I20260806 06:26:38.397250 25799 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.190:41799 every 8 connection(s)
I20260806 06:26:38.397578 25699 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785997581049705-21554-0/minicluster-data/master-0/data/info.pb
I20260806 06:26:38.398538 25801 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 06:26:38.400686 25801 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95aa6306150c4ed087da78c36c9bd59f: Bootstrap starting.
I20260806 06:26:38.401196 25801 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 95aa6306150c4ed087da78c36c9bd59f: Neither blocks nor log segments found. Creating new log.
I20260806 06:26:38.401424 25801 log.cc:824] T 00000000000000000000000000000000 P 95aa6306150c4ed087da78c36c9bd59f: Log is configured to *not* fsync() on all Append() calls
I20260806 06:26:38.402010 25801 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95aa6306150c4ed087da78c36c9bd59f: No bootstrap required, opened a new log
I20260806 06:26:38.403586 25801 raft_consensus.cc:348] T 00000000000000000000000000000000 P 95aa6306150c4ed087da78c36c9bd59f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95aa6306150c4ed087da78c36c9bd59f" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 41799 } }
I20260806 06:26:38.403721 25801 raft_consensus.cc:374] T 00000000000000000000000000000000 P 95aa6306150c4ed087da78c36c9bd59f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 06:26:38.403743 25801 raft_consensus.cc:729] T 00000000000000000000000000000000 P 95aa6306150c4ed087da78c36c9bd59f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95aa6306150c4ed087da78c36c9bd59f, State: Initialized, Role: FOLLOWER
I20260806 06:26:38.403851 25801 consensus_queue.cc:260] T 00000000000000000000000000000000 P 95aa6306150c4ed087da78c36c9bd59f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95aa6306150c4ed087da78c36c9bd59f" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 41799 } }
I20260806 06:26:38.403927 25801 raft_consensus.cc:388] T 00000000000000000000000000000000 P 95aa6306150c4ed087da78c36c9bd59f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 06:26:38.403950 25801 raft_consensus.cc:482] T 00000000000000000000000000000000 P 95aa6306150c4ed087da78c36c9bd59f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 06:26:38.403976 25801 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 95aa6306150c4ed087da78c36c9bd59f [term 0 FOLLOWER]: Advancing to term 1
I20260806 06:26:38.404706 25801 raft_consensus.cc:504] T 00000000000000000000000000000000 P 95aa6306150c4ed087da78c36c9bd59f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95aa6306150c4ed087da78c36c9bd59f" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 41799 } }
I20260806 06:26:38.404793 25801 leader_election.cc:302] T 00000000000000000000000000000000 P 95aa6306150c4ed087da78c36c9bd59f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 95aa6306150c4ed087da78c36c9bd59f; no voters:
I20260806 06:26:38.404913 25801 leader_election.cc:288] T 00000000000000000000000000000000 P 95aa6306150c4ed087da78c36c9bd59f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 06:26:38.404958 25806 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 95aa6306150c4ed087da78c36c9bd59f [term 1 FOLLOWER]: Leader election won for term 1
I20260806 06:26:38.405128 25801 sys_catalog.cc:564] T 00000000000000000000000000000000 P 95aa6306150c4ed087da78c36c9bd59f [sys.catalog]: configured and running, proceeding with master startup.
I20260806 06:26:38.405165 25806 raft_consensus.cc:686] T 00000000000000000000000000000000 P 95aa6306150c4ed087da78c36c9bd59f [term 1 LEADER]: Becoming Leader. State: Replica: 95aa6306150c4ed087da78c36c9bd59f, State: Running, Role: LEADER
I20260806 06:26:38.405329 25806 consensus_queue.cc:237] T 00000000000000000000000000000000 P 95aa6306150c4ed087da78c36c9bd59f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95aa6306150c4ed087da78c36c9bd59f" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 41799 } }
I20260806 06:26:38.405750 25808 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95aa6306150c4ed087da78c36c9bd59f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 95aa6306150c4ed087da78c36c9bd59f. Latest consensus state: current_term: 1 leader_uuid: "95aa6306150c4ed087da78c36c9bd59f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95aa6306150c4ed087da78c36c9bd59f" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 41799 } } }
I20260806 06:26:38.405959 25808 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95aa6306150c4ed087da78c36c9bd59f [sys.catalog]: This master's current role is: LEADER
I20260806 06:26:38.405895 25807 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95aa6306150c4ed087da78c36c9bd59f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "95aa6306150c4ed087da78c36c9bd59f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95aa6306150c4ed087da78c36c9bd59f" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 41799 } } }
I20260806 06:26:38.406795 25807 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95aa6306150c4ed087da78c36c9bd59f [sys.catalog]: This master's current role is: LEADER
W20260806 06:26:38.407179 25830 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 95aa6306150c4ed087da78c36c9bd59f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 06:26:38.407261 25830 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 06:26:38.407316 25831 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 06:26:38.407481 25831 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 06:26:38.407701 21554 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHhLhT/build/release/bin/kudu as pid 25699
I20260806 06:26:38.407807 21554 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785997581049705-21554-0/minicluster-data/master-0/wal/instance
I20260806 06:26:38.407979 21554 external_mini_cluster.cc:1428] Setting key 8655fedf6da8a71bc89f2abd035534d5
I20260806 06:26:38.409484 25831 catalog_manager.cc:1347] Generated new cluster ID: 875a87a14a1143868cb9a4e0a7605b75
I20260806 06:26:38.409534 25831 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 06 06:26:38 dist-test-slave-4tdj krb5kdc[25683](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997598, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.12.190@KRBTEST.COM
I20260806 06:26:38.419095 25834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:38.408603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44148 (local address 127.21.12.190:41799)
0806 06:26:38.408741 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0806 06:26:38.408751 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 06:26:38.408780 (+ 29us) server_negotiation.cc:407] Connection header received
0806 06:26:38.408876 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:26:38.408881 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:26:38.408933 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:26:38.409811 (+ 878us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:26:38.409898 (+ 87us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:38.411269 (+ 1371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:38.412288 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:38.412888 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:38.413244 (+ 356us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:38.415670 (+ 2426us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:26:38.415674 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:26:38.415675 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 06:26:38.415686 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:26:38.417512 (+ 1826us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:26:38.417891 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:26:38.417893 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:26:38.417894 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:26:38.417990 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:26:38.418169 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:26:38.418170 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:26:38.418170 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:26:38.418478 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:26:38.418618 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0806 06:26:38.418735 (+ 117us) server_negotiation.cc:299] Negotiation successful
0806 06:26:38.418867 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95,"thread_start_us":71,"threads_started":1}
I20260806 06:26:38.419749 25831 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 06:26:38.420279 25831 catalog_manager.cc:1504] Loading token signing keys...
I20260806 06:26:38.429631 25831 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 95aa6306150c4ed087da78c36c9bd59f: Generated new TSK 0
I20260806 06:26:38.429817 25831 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.12.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.12.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.12.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.129.keytab.
Entry for principal kudu/127.21.12.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.129.keytab.
I20260806 06:26:38.460378 21554 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHhLhT/build/release/bin/kudu
/tmp/dist-test-taskFHhLhT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785997581049705-21554-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785997581049705-21554-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785997581049705-21554-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785997581049705-21554-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.12.129:0
--local_ip_for_outbound_sockets=127.21.12.129
--webserver_interface=127.21.12.129
--webserver_port=0
--tserver_master_addrs=127.21.12.190:41799
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.129.keytab
--principal=kudu/127.21.12.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.148:35701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785997581049705-21554-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785997581049705-21554-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785997581049705-21554-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 06:26:38.540771 25839 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 06:26:38.540970 25839 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 06:26:38.540993 25839 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 06:26:38.542114 25839 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 06:26:38.542152 25839 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 06:26:38.542176 25839 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.12.129
I20260806 06:26:38.543617 25839 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.148:35701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785997581049705-21554-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785997581049705-21554-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.129.keytab
--principal=kudu/127.21.12.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.12.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785997581049705-21554-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.12.129
--webserver_port=0
--tserver_master_addrs=127.21.12.190:41799
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25839
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.12.129
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785997581049705-21554-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 05:11:02 UTC on ccycloud.quasar-hvzbbz.root.comops.site
build id 4569617
I20260806 06:26:38.543826 25839 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 06:26:38.544044 25839 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 06:26:38.548838 25844 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:26:38.548962 25839 server_base.cc:1034] running on GCE node
W20260806 06:26:38.548856 25846 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:26:38.548869 25850 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:26:38.549270 25839 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 06:26:38.549492 25839 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 06:26:38.550638 25839 hybrid_clock.cc:648] HybridClock initialized: now 1785997598550626 us; error 30 us; skew 500 ppm
Aug 06 06:26:38 dist-test-slave-4tdj krb5kdc[25683](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997598, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 06:26:38.553027 25839 init.cc:377] Logged in from keytab as kudu/127.21.12.129@KRBTEST.COM (short username kudu)
I20260806 06:26:38.553427 25839 webserver.cc:460] Webserver started at http://127.21.12.129:45399/ using document root <none> and password file <none>
I20260806 06:26:38.553622 25839 fs_manager.cc:359] Metadata directory not provided
I20260806 06:26:38.553665 25839 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 06:26:38.553778 25839 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 06:26:38.554823 25839 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785997581049705-21554-0/minicluster-data/ts-0/data/instance:
uuid: "f8e32be8555540d78a1c0f2c6eb53a62"
format_stamp: "Formatted at 2026-08-06 06:26:38 on dist-test-slave-4tdj"
server_key: "9c6679f821be792f7b65997e3ef67b85"
server_key_iv: "f8f2f55cdcd60652f09dbce269ca4b03"
server_key_version: "encryptionkey@0"
I20260806 06:26:38.555157 25839 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785997581049705-21554-0/minicluster-data/ts-0/wal/instance:
uuid: "f8e32be8555540d78a1c0f2c6eb53a62"
format_stamp: "Formatted at 2026-08-06 06:26:38 on dist-test-slave-4tdj"
server_key: "9c6679f821be792f7b65997e3ef67b85"
server_key_iv: "f8f2f55cdcd60652f09dbce269ca4b03"
server_key_version: "encryptionkey@0"
I20260806 06:26:38.556334 25839 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260806 06:26:38.556994 25860 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 06:26:38.557144 25839 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260806 06:26:38.557219 25839 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785997581049705-21554-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785997581049705-21554-0/minicluster-data/ts-0/wal
uuid: "f8e32be8555540d78a1c0f2c6eb53a62"
format_stamp: "Formatted at 2026-08-06 06:26:38 on dist-test-slave-4tdj"
server_key: "9c6679f821be792f7b65997e3ef67b85"
server_key_iv: "f8f2f55cdcd60652f09dbce269ca4b03"
server_key_version: "encryptionkey@0"
I20260806 06:26:38.557282 25839 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785997581049705-21554-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785997581049705-21554-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785997581049705-21554-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 06:26:38.575691 25839 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 06:26:38.577576 25839 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 06:26:38.577700 25839 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 06:26:38.577908 25839 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 06:26:38.578207 25839 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 06:26:38.578241 25839 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 06:26:38.578308 25839 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 06:26:38.578336 25839 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 06:26:38.583909 25839 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.129:39023
I20260806 06:26:38.583974 26020 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.129:39023 every 8 connection(s)
I20260806 06:26:38.584254 25839 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785997581049705-21554-0/minicluster-data/ts-0/data/info.pb
I20260806 06:26:38.585484 21554 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHhLhT/build/release/bin/kudu as pid 25839
I20260806 06:26:38.585587 21554 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785997581049705-21554-0/minicluster-data/ts-0/wal/instance
I20260806 06:26:38.585788 21554 external_mini_cluster.cc:1428] Setting key b64c53d20b945305514fb35414dc51af
Aug 06 06:26:38 dist-test-slave-4tdj krb5kdc[25683](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997598, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.129@KRBTEST.COM for kudu/127.21.12.190@KRBTEST.COM
I20260806 06:26:38.594013 25834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:38.584939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.129:38923 (local address 127.21.12.190:41799)
0806 06:26:38.584997 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0806 06:26:38.585000 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 06:26:38.585195 (+ 195us) server_negotiation.cc:407] Connection header received
0806 06:26:38.585861 (+ 666us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:26:38.585862 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:26:38.585877 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:26:38.585924 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:26:38.587097 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:38.587603 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:38.588743 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:38.588947 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:38.589013 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:38.591297 (+ 2284us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:26:38.591299 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:26:38.591299 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 06:26:38.591305 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:26:38.592533 (+ 1228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:26:38.593015 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:26:38.593016 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:26:38.593016 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:26:38.593077 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:26:38.593349 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:26:38.593349 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:26:38.593350 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:26:38.593482 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:26:38.593541 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0806 06:26:38.593863 (+ 322us) server_negotiation.cc:299] Negotiation successful
0806 06:26:38.593939 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260806 06:26:38.594233 26023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:38.584911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:41799 (local address 127.21.12.129:38923)
0806 06:26:38.585078 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0806 06:26:38.585119 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0806 06:26:38.585738 (+ 619us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:26:38.585999 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:26:38.586003 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:26:38.586737 (+ 734us) client_negotiation.cc:189] Negotiated authn=SASL
0806 06:26:38.586987 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:38.586993 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:38.587686 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:38.587687 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:38.588660 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:38.588663 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:38.589017 (+ 354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:38.589018 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:38.589145 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:38.589599 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 06:26:38.589607 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0806 06:26:38.591207 (+ 1600us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 06:26:38.592619 (+ 1412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:26:38.592627 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:26:38.592630 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:26:38.592920 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:26:38.593145 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:26:38.593146 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:26:38.593146 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:26:38.593277 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:26:38.593552 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 06:26:38.593559 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 06:26:38.593729 (+ 170us) client_negotiation.cc:769] Sending connection context
0806 06:26:38.593849 (+ 120us) client_negotiation.cc:240] Negotiation successful
0806 06:26:38.594013 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":94,"threads_started":1}
WARNING: no policy specified for kudu/127.21.12.130@KRBTEST.COM; defaulting to no policy
I20260806 06:26:38.594792 26021 heartbeater.cc:347] Connected to a master server at 127.21.12.190:41799
I20260806 06:26:38.594877 26021 heartbeater.cc:464] Registering TS with master...
I20260806 06:26:38.595041 26021 heartbeater.cc:511] Master 127.21.12.190:41799 requested a full tablet report, sending...
I20260806 06:26:38.595520 25744 ts_manager.cc:194] Registered new tserver with Master: f8e32be8555540d78a1c0f2c6eb53a62 (127.21.12.129:39023)
I20260806 06:26:38.596083 25744 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.12.129@KRBTEST.COM'} at 127.21.12.129:38923
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.12.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.12.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.130.keytab.
Entry for principal kudu/127.21.12.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.130.keytab.
I20260806 06:26:38.611105 21554 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHhLhT/build/release/bin/kudu
/tmp/dist-test-taskFHhLhT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785997581049705-21554-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785997581049705-21554-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785997581049705-21554-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785997581049705-21554-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.12.130:0
--local_ip_for_outbound_sockets=127.21.12.130
--webserver_interface=127.21.12.130
--webserver_port=0
--tserver_master_addrs=127.21.12.190:41799
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.130.keytab
--principal=kudu/127.21.12.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.148:35701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785997581049705-21554-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785997581049705-21554-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785997581049705-21554-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 06:26:38.690754 26027 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 06:26:38.690946 26027 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 06:26:38.690968 26027 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 06:26:38.692095 26027 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 06:26:38.692137 26027 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 06:26:38.692162 26027 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.12.130
I20260806 06:26:38.693576 26027 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.148:35701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785997581049705-21554-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785997581049705-21554-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.130.keytab
--principal=kudu/127.21.12.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.12.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785997581049705-21554-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.12.130
--webserver_port=0
--tserver_master_addrs=127.21.12.190:41799
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26027
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.12.130
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785997581049705-21554-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 05:11:02 UTC on ccycloud.quasar-hvzbbz.root.comops.site
build id 4569617
I20260806 06:26:38.693804 26027 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 06:26:38.694008 26027 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 06:26:38.698881 26038 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:26:38.698912 26036 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:26:38.698921 26035 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:26:38.698956 26027 server_base.cc:1034] running on GCE node
I20260806 06:26:38.699335 26027 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 06:26:38.699585 26027 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 06:26:38.700738 26027 hybrid_clock.cc:648] HybridClock initialized: now 1785997598700724 us; error 29 us; skew 500 ppm
Aug 06 06:26:38 dist-test-slave-4tdj krb5kdc[25683](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997598, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 06:26:38.703191 26027 init.cc:377] Logged in from keytab as kudu/127.21.12.130@KRBTEST.COM (short username kudu)
I20260806 06:26:38.703656 26027 webserver.cc:460] Webserver started at http://127.21.12.130:43765/ using document root <none> and password file <none>
I20260806 06:26:38.703858 26027 fs_manager.cc:359] Metadata directory not provided
I20260806 06:26:38.703905 26027 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 06:26:38.704031 26027 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 06:26:38.705090 26027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785997581049705-21554-0/minicluster-data/ts-1/data/instance:
uuid: "a551ffcc6a834034b3f8f2ceb588ba80"
format_stamp: "Formatted at 2026-08-06 06:26:38 on dist-test-slave-4tdj"
server_key: "c0bf6203f8ebe9b640623b2ce19adfc8"
server_key_iv: "b6a621c8f1bd411c65bceeb47838edac"
server_key_version: "encryptionkey@0"
I20260806 06:26:38.705477 26027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785997581049705-21554-0/minicluster-data/ts-1/wal/instance:
uuid: "a551ffcc6a834034b3f8f2ceb588ba80"
format_stamp: "Formatted at 2026-08-06 06:26:38 on dist-test-slave-4tdj"
server_key: "c0bf6203f8ebe9b640623b2ce19adfc8"
server_key_iv: "b6a621c8f1bd411c65bceeb47838edac"
server_key_version: "encryptionkey@0"
I20260806 06:26:38.706889 26027 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260806 06:26:38.707613 26048 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 06:26:38.707764 26027 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260806 06:26:38.707851 26027 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785997581049705-21554-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785997581049705-21554-0/minicluster-data/ts-1/wal
uuid: "a551ffcc6a834034b3f8f2ceb588ba80"
format_stamp: "Formatted at 2026-08-06 06:26:38 on dist-test-slave-4tdj"
server_key: "c0bf6203f8ebe9b640623b2ce19adfc8"
server_key_iv: "b6a621c8f1bd411c65bceeb47838edac"
server_key_version: "encryptionkey@0"
I20260806 06:26:38.707911 26027 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785997581049705-21554-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785997581049705-21554-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785997581049705-21554-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 06:26:38.720788 26027 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 06:26:38.722680 26027 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 06:26:38.722818 26027 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 06:26:38.723030 26027 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 06:26:38.723318 26027 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 06:26:38.723351 26027 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 06:26:38.723378 26027 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 06:26:38.723402 26027 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 06:26:38.728891 26027 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.130:43375
I20260806 06:26:38.728932 26211 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.130:43375 every 8 connection(s)
I20260806 06:26:38.729231 26027 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785997581049705-21554-0/minicluster-data/ts-1/data/info.pb
Aug 06 06:26:38 dist-test-slave-4tdj krb5kdc[25683](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997598, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.130@KRBTEST.COM for kudu/127.21.12.190@KRBTEST.COM
I20260806 06:26:38.735776 21554 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHhLhT/build/release/bin/kudu as pid 26027
I20260806 06:26:38.735885 21554 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785997581049705-21554-0/minicluster-data/ts-1/wal/instance
I20260806 06:26:38.736078 21554 external_mini_cluster.cc:1428] Setting key ea954829d2c1c39c6a481106cbb0f5e2
I20260806 06:26:38.738031 25834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:38.729943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.130:50593 (local address 127.21.12.190:41799)
0806 06:26:38.730006 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0806 06:26:38.730008 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:26:38.730201 (+ 193us) server_negotiation.cc:407] Connection header received
0806 06:26:38.730732 (+ 531us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:26:38.730733 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:26:38.730742 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:26:38.730794 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:26:38.731690 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:38.732179 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:38.733109 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:38.733288 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:38.733335 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:38.735381 (+ 2046us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:26:38.735383 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:26:38.735383 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 06:26:38.735390 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:26:38.736799 (+ 1409us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:26:38.737184 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:26:38.737186 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:26:38.737186 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:26:38.737253 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:26:38.737450 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:26:38.737451 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:26:38.737451 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:26:38.737577 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:26:38.737649 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0806 06:26:38.737854 (+ 205us) server_negotiation.cc:299] Negotiation successful
0806 06:26:38.737936 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260806 06:26:38.738095 26216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:38.729908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:41799 (local address 127.21.12.130:50593)
0806 06:26:38.730073 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0806 06:26:38.730117 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0806 06:26:38.730646 (+ 529us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:26:38.730854 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:26:38.730858 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:26:38.731420 (+ 562us) client_negotiation.cc:189] Negotiated authn=SASL
0806 06:26:38.731622 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:38.731626 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:38.732239 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:38.732240 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:38.733041 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:38.733044 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:38.733357 (+ 313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:38.733358 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:38.733461 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:38.733959 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 06:26:38.733967 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0806 06:26:38.735293 (+ 1326us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 06:26:38.736885 (+ 1592us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:26:38.736891 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:26:38.736894 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:26:38.737118 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:26:38.737315 (+ 197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:26:38.737315 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:26:38.737315 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:26:38.737403 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:26:38.737637 (+ 234us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 06:26:38.737641 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 06:26:38.737747 (+ 106us) client_negotiation.cc:769] Sending connection context
0806 06:26:38.737822 (+ 75us) client_negotiation.cc:240] Negotiation successful
0806 06:26:38.737935 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":76,"threads_started":1}
I20260806 06:26:38.738811 26212 heartbeater.cc:347] Connected to a master server at 127.21.12.190:41799
I20260806 06:26:38.738904 26212 heartbeater.cc:464] Registering TS with master...
I20260806 06:26:38.739077 26212 heartbeater.cc:511] Master 127.21.12.190:41799 requested a full tablet report, sending...
I20260806 06:26:38.739449 25744 ts_manager.cc:194] Registered new tserver with Master: a551ffcc6a834034b3f8f2ceb588ba80 (127.21.12.130:43375)
I20260806 06:26:38.739873 25744 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.12.130@KRBTEST.COM'} at 127.21.12.130:50593
WARNING: no policy specified for kudu/127.21.12.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.12.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.12.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.131.keytab.
Entry for principal kudu/127.21.12.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.131.keytab.
I20260806 06:26:38.761400 21554 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHhLhT/build/release/bin/kudu
/tmp/dist-test-taskFHhLhT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785997581049705-21554-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785997581049705-21554-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785997581049705-21554-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785997581049705-21554-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.12.131:0
--local_ip_for_outbound_sockets=127.21.12.131
--webserver_interface=127.21.12.131
--webserver_port=0
--tserver_master_addrs=127.21.12.190:41799
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.131.keytab
--principal=kudu/127.21.12.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.148:35701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785997581049705-21554-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785997581049705-21554-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785997581049705-21554-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 06:26:38.842725 26219 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 06:26:38.842929 26219 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 06:26:38.842953 26219 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 06:26:38.844023 26219 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 06:26:38.844062 26219 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 06:26:38.844089 26219 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.12.131
I20260806 06:26:38.845485 26219 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.148:35701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785997581049705-21554-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785997581049705-21554-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.131.keytab
--principal=kudu/127.21.12.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.12.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785997581049705-21554-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.12.131
--webserver_port=0
--tserver_master_addrs=127.21.12.190:41799
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26219
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.12.131
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785997581049705-21554-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 05:11:02 UTC on ccycloud.quasar-hvzbbz.root.comops.site
build id 4569617
I20260806 06:26:38.845665 26219 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 06:26:38.845876 26219 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 06:26:38.850643 26225 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:26:38.850652 26226 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:26:38.850652 26230 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:26:38.850708 26219 server_base.cc:1034] running on GCE node
I20260806 06:26:38.851032 26219 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 06:26:38.851248 26219 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 06:26:38.852383 26219 hybrid_clock.cc:648] HybridClock initialized: now 1785997598852369 us; error 31 us; skew 500 ppm
Aug 06 06:26:38 dist-test-slave-4tdj krb5kdc[25683](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997598, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 06:26:38.858127 26219 init.cc:377] Logged in from keytab as kudu/127.21.12.131@KRBTEST.COM (short username kudu)
I20260806 06:26:38.858616 26219 webserver.cc:460] Webserver started at http://127.21.12.131:32995/ using document root <none> and password file <none>
I20260806 06:26:38.858816 26219 fs_manager.cc:359] Metadata directory not provided
I20260806 06:26:38.858862 26219 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 06:26:38.858999 26219 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 06:26:38.860086 26219 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785997581049705-21554-0/minicluster-data/ts-2/data/instance:
uuid: "69b602f1513e4e438dd8f2d51f811e88"
format_stamp: "Formatted at 2026-08-06 06:26:38 on dist-test-slave-4tdj"
server_key: "88297355824034e982a370c8b5040ee5"
server_key_iv: "947965d62abe9b163dde99c6c567ef30"
server_key_version: "encryptionkey@0"
I20260806 06:26:38.860447 26219 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785997581049705-21554-0/minicluster-data/ts-2/wal/instance:
uuid: "69b602f1513e4e438dd8f2d51f811e88"
format_stamp: "Formatted at 2026-08-06 06:26:38 on dist-test-slave-4tdj"
server_key: "88297355824034e982a370c8b5040ee5"
server_key_iv: "947965d62abe9b163dde99c6c567ef30"
server_key_version: "encryptionkey@0"
I20260806 06:26:38.861783 26219 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260806 06:26:38.862562 26237 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 06:26:38.862715 26219 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260806 06:26:38.862804 26219 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785997581049705-21554-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785997581049705-21554-0/minicluster-data/ts-2/wal
uuid: "69b602f1513e4e438dd8f2d51f811e88"
format_stamp: "Formatted at 2026-08-06 06:26:38 on dist-test-slave-4tdj"
server_key: "88297355824034e982a370c8b5040ee5"
server_key_iv: "947965d62abe9b163dde99c6c567ef30"
server_key_version: "encryptionkey@0"
I20260806 06:26:38.862874 26219 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785997581049705-21554-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785997581049705-21554-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785997581049705-21554-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 06:26:38.900391 26219 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 06:26:38.902500 26219 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 06:26:38.902648 26219 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 06:26:38.902901 26219 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 06:26:38.903226 26219 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 06:26:38.903264 26219 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 06:26:38.903291 26219 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 06:26:38.903318 26219 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 06:26:38.909799 26219 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.131:39869
I20260806 06:26:38.909842 26397 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.131:39869 every 8 connection(s)
I20260806 06:26:38.910158 26219 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785997581049705-21554-0/minicluster-data/ts-2/data/info.pb
Aug 06 06:26:38 dist-test-slave-4tdj krb5kdc[25683](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997598, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.131@KRBTEST.COM for kudu/127.21.12.190@KRBTEST.COM
I20260806 06:26:38.916812 21554 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHhLhT/build/release/bin/kudu as pid 26219
I20260806 06:26:38.916908 21554 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785997581049705-21554-0/minicluster-data/ts-2/wal/instance
I20260806 06:26:38.917109 21554 external_mini_cluster.cc:1428] Setting key a203597fa86a1ec3a8895ae29f2e24cf
I20260806 06:26:38.919298 25834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:38.910846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.131:49011 (local address 127.21.12.190:41799)
0806 06:26:38.910891 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0806 06:26:38.910893 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:26:38.911115 (+ 222us) server_negotiation.cc:407] Connection header received
0806 06:26:38.911671 (+ 556us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:26:38.911672 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:26:38.911684 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:26:38.911741 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:26:38.912699 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:38.913221 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:38.914201 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:38.914429 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:38.914484 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:38.916433 (+ 1949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:26:38.916434 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:26:38.916435 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 06:26:38.916442 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:26:38.917973 (+ 1531us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:26:38.918372 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:26:38.918374 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:26:38.918374 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:26:38.918439 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:26:38.918675 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:26:38.918675 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:26:38.918676 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:26:38.918796 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:26:38.918861 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0806 06:26:38.919168 (+ 307us) server_negotiation.cc:299] Negotiation successful
0806 06:26:38.919235 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260806 06:26:38.919535 26404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:38.910824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:41799 (local address 127.21.12.131:49011)
0806 06:26:38.910996 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0806 06:26:38.911039 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0806 06:26:38.911535 (+ 496us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:26:38.911814 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:26:38.911817 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:26:38.912385 (+ 568us) client_negotiation.cc:189] Negotiated authn=SASL
0806 06:26:38.912585 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:38.912590 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:38.913311 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:38.913312 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:38.914116 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:38.914119 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:38.914515 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:38.914516 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:38.914626 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:38.915020 (+ 394us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 06:26:38.915028 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0806 06:26:38.916338 (+ 1310us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 06:26:38.918040 (+ 1702us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:26:38.918047 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:26:38.918050 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:26:38.918274 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:26:38.918506 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:26:38.918507 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:26:38.918507 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:26:38.918615 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:26:38.918857 (+ 242us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 06:26:38.918864 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 06:26:38.919012 (+ 148us) client_negotiation.cc:769] Sending connection context
0806 06:26:38.919137 (+ 125us) client_negotiation.cc:240] Negotiation successful
0806 06:26:38.919304 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":75,"threads_started":1}
I20260806 06:26:38.919981 26400 heartbeater.cc:347] Connected to a master server at 127.21.12.190:41799
I20260806 06:26:38.920080 26400 heartbeater.cc:464] Registering TS with master...
I20260806 06:26:38.920228 26400 heartbeater.cc:511] Master 127.21.12.190:41799 requested a full tablet report, sending...
I20260806 06:26:38.920610 25744 ts_manager.cc:194] Registered new tserver with Master: 69b602f1513e4e438dd8f2d51f811e88 (127.21.12.131:39869)
I20260806 06:26:38.920941 25744 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.12.131@KRBTEST.COM'} at 127.21.12.131:49011
I20260806 06:26:38.928828 21554 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 06 06:26:38 dist-test-slave-4tdj krb5kdc[25683](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997598, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Aug 06 06:26:38 dist-test-slave-4tdj krb5kdc[25683](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997598, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.21.12.190@KRBTEST.COM
I20260806 06:26:38.950904 25834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:38.944414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44156 (local address 127.21.12.190:41799)
0806 06:26:38.944498 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0806 06:26:38.944500 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:26:38.944578 (+ 78us) server_negotiation.cc:407] Connection header received
0806 06:26:38.944621 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:26:38.944622 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:26:38.944633 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:26:38.944685 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:26:38.945311 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:38.945803 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:38.946593 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:38.946776 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:38.946827 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:38.948837 (+ 2010us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:26:38.948839 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:26:38.948839 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 06:26:38.948846 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:26:38.949981 (+ 1135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:26:38.950265 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:26:38.950266 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:26:38.950266 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:26:38.950365 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:26:38.950502 (+ 137us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:26:38.950502 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:26:38.950503 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:26:38.950638 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:26:38.950705 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0806 06:26:38.950781 (+ 76us) server_negotiation.cc:299] Negotiation successful
0806 06:26:38.950836 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
W20260806 06:26:38.951148 25744 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:44156
I20260806 06:26:38.951596 21554 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHhLhT/build/release/bin/kudu with pid 25839
I20260806 06:26:38.960667 21554 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHhLhT/build/release/bin/kudu with pid 26027
I20260806 06:26:38.966477 21554 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHhLhT/build/release/bin/kudu with pid 26219
I20260806 06:26:38.973346 21554 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHhLhT/build/release/bin/kudu with pid 25699
2026-08-06T06:26:38Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (1917 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785997581049705-21554-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 06:26:39 dist-test-slave-4tdj krb5kdc[26422](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 06:26:39 dist-test-slave-4tdj krb5kdc[26422](info): set up 2 sockets
Aug 06 06:26:39 dist-test-slave-4tdj krb5kdc[26422](info): commencing operation
krb5kdc: starting...
W20260806 06:26:40.101512 21554 mini_kdc.cc:121] Time spent starting KDC: real 1.105s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785997581049705-21554-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785997581049705-21554-0/krb5kdc/test-user.keytab.
Aug 06 06:26:40 dist-test-slave-4tdj krb5kdc[26422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997600, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T06:26:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T06:26:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.12.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.12.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.12.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.190.keytab.
Entry for principal kudu/127.21.12.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.190.keytab.
I20260806 06:26:40.221022 21554 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHhLhT/build/release/bin/kudu
/tmp/dist-test-taskFHhLhT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785997581049705-21554-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785997581049705-21554-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785997581049705-21554-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785997581049705-21554-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.12.190:39729
--webserver_interface=127.21.12.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.190.keytab
--principal=kudu/127.21.12.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.148:45671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.12.190:39729
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785997581049705-21554-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785997581049705-21554-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785997581049705-21554-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 06:26:40.300151 26439 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 06:26:40.300338 26439 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 06:26:40.300366 26439 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 06:26:40.301455 26439 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 06:26:40.301493 26439 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 06:26:40.301519 26439 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 06:26:40.301535 26439 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 06:26:40.301559 26439 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 06:26:40.303067 26439 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.148:45671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785997581049705-21554-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785997581049705-21554-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.12.190:39729
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.190.keytab
--principal=kudu/127.21.12.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.12.190:39729
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785997581049705-21554-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.12.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26439
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785997581049705-21554-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 05:11:02 UTC on ccycloud.quasar-hvzbbz.root.comops.site
build id 4569617
I20260806 06:26:40.303284 26439 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 06:26:40.303476 26439 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 06:26:40.308159 26454 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:26:40.308321 26439 server_base.cc:1034] running on GCE node
W20260806 06:26:40.308162 26449 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:26:40.308179 26445 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:26:40.308660 26439 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 06:26:40.308914 26439 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 06:26:40.310055 26439 hybrid_clock.cc:648] HybridClock initialized: now 1785997600310038 us; error 34 us; skew 500 ppm
Aug 06 06:26:40 dist-test-slave-4tdj krb5kdc[26422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997600, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 06:26:40.312489 26439 init.cc:377] Logged in from keytab as kudu/127.21.12.190@KRBTEST.COM (short username kudu)
I20260806 06:26:40.312913 26439 webserver.cc:460] Webserver started at http://127.21.12.190:45749/ using document root <none> and password file <none>
I20260806 06:26:40.313143 26439 fs_manager.cc:359] Metadata directory not provided
I20260806 06:26:40.313187 26439 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 06:26:40.313302 26439 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 06:26:40.314364 26439 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785997581049705-21554-0/minicluster-data/master-0/data/instance:
uuid: "ed06f54ab7ea4463a0edc5b64e7fd41d"
format_stamp: "Formatted at 2026-08-06 06:26:40 on dist-test-slave-4tdj"
server_key: "d644e313e135b28600bfa5622c8280b4"
server_key_iv: "ee585c362daaf9ab8754ee0fdb310267"
server_key_version: "encryptionkey@0"
I20260806 06:26:40.314700 26439 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785997581049705-21554-0/minicluster-data/master-0/wal/instance:
uuid: "ed06f54ab7ea4463a0edc5b64e7fd41d"
format_stamp: "Formatted at 2026-08-06 06:26:40 on dist-test-slave-4tdj"
server_key: "d644e313e135b28600bfa5622c8280b4"
server_key_iv: "ee585c362daaf9ab8754ee0fdb310267"
server_key_version: "encryptionkey@0"
I20260806 06:26:40.315915 26439 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260806 06:26:40.316660 26466 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 06:26:40.316808 26439 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260806 06:26:40.316883 26439 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785997581049705-21554-0/minicluster-data/master-0/data,/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785997581049705-21554-0/minicluster-data/master-0/wal
uuid: "ed06f54ab7ea4463a0edc5b64e7fd41d"
format_stamp: "Formatted at 2026-08-06 06:26:40 on dist-test-slave-4tdj"
server_key: "d644e313e135b28600bfa5622c8280b4"
server_key_iv: "ee585c362daaf9ab8754ee0fdb310267"
server_key_version: "encryptionkey@0"
I20260806 06:26:40.316947 26439 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785997581049705-21554-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785997581049705-21554-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785997581049705-21554-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 06:26:40.339133 26439 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 06:26:40.341204 26439 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 06:26:40.341323 26439 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 06:26:40.345276 26439 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.190:39729
I20260806 06:26:40.345330 26553 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.190:39729 every 8 connection(s)
I20260806 06:26:40.345785 26439 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785997581049705-21554-0/minicluster-data/master-0/data/info.pb
I20260806 06:26:40.345968 21554 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHhLhT/build/release/bin/kudu as pid 26439
I20260806 06:26:40.346073 21554 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785997581049705-21554-0/minicluster-data/master-0/wal/instance
I20260806 06:26:40.346262 21554 external_mini_cluster.cc:1428] Setting key fc6ec939cb1f98ac2a958f4806a8aa9e
I20260806 06:26:40.346550 26554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 06:26:40.349699 26554 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed06f54ab7ea4463a0edc5b64e7fd41d: Bootstrap starting.
I20260806 06:26:40.350359 26554 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ed06f54ab7ea4463a0edc5b64e7fd41d: Neither blocks nor log segments found. Creating new log.
I20260806 06:26:40.350646 26554 log.cc:824] T 00000000000000000000000000000000 P ed06f54ab7ea4463a0edc5b64e7fd41d: Log is configured to *not* fsync() on all Append() calls
I20260806 06:26:40.351372 26554 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed06f54ab7ea4463a0edc5b64e7fd41d: No bootstrap required, opened a new log
Aug 06 06:26:40 dist-test-slave-4tdj krb5kdc[26422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997600, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.12.190@KRBTEST.COM
I20260806 06:26:40.353435 26554 raft_consensus.cc:348] T 00000000000000000000000000000000 P ed06f54ab7ea4463a0edc5b64e7fd41d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed06f54ab7ea4463a0edc5b64e7fd41d" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 39729 } }
I20260806 06:26:40.353596 26554 raft_consensus.cc:374] T 00000000000000000000000000000000 P ed06f54ab7ea4463a0edc5b64e7fd41d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 06:26:40.353641 26554 raft_consensus.cc:729] T 00000000000000000000000000000000 P ed06f54ab7ea4463a0edc5b64e7fd41d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed06f54ab7ea4463a0edc5b64e7fd41d, State: Initialized, Role: FOLLOWER
I20260806 06:26:40.353782 26554 consensus_queue.cc:260] T 00000000000000000000000000000000 P ed06f54ab7ea4463a0edc5b64e7fd41d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed06f54ab7ea4463a0edc5b64e7fd41d" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 39729 } }
I20260806 06:26:40.353883 26554 raft_consensus.cc:388] T 00000000000000000000000000000000 P ed06f54ab7ea4463a0edc5b64e7fd41d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 06:26:40.353921 26554 raft_consensus.cc:482] T 00000000000000000000000000000000 P ed06f54ab7ea4463a0edc5b64e7fd41d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 06:26:40.353971 26554 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ed06f54ab7ea4463a0edc5b64e7fd41d [term 0 FOLLOWER]: Advancing to term 1
I20260806 06:26:40.354674 26554 raft_consensus.cc:504] T 00000000000000000000000000000000 P ed06f54ab7ea4463a0edc5b64e7fd41d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed06f54ab7ea4463a0edc5b64e7fd41d" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 39729 } }
I20260806 06:26:40.354786 26554 leader_election.cc:302] T 00000000000000000000000000000000 P ed06f54ab7ea4463a0edc5b64e7fd41d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ed06f54ab7ea4463a0edc5b64e7fd41d; no voters:
I20260806 06:26:40.354947 26554 leader_election.cc:288] T 00000000000000000000000000000000 P ed06f54ab7ea4463a0edc5b64e7fd41d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 06:26:40.354992 26565 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ed06f54ab7ea4463a0edc5b64e7fd41d [term 1 FOLLOWER]: Leader election won for term 1
I20260806 06:26:40.355151 26565 raft_consensus.cc:686] T 00000000000000000000000000000000 P ed06f54ab7ea4463a0edc5b64e7fd41d [term 1 LEADER]: Becoming Leader. State: Replica: ed06f54ab7ea4463a0edc5b64e7fd41d, State: Running, Role: LEADER
I20260806 06:26:40.355233 26554 sys_catalog.cc:564] T 00000000000000000000000000000000 P ed06f54ab7ea4463a0edc5b64e7fd41d [sys.catalog]: configured and running, proceeding with master startup.
I20260806 06:26:40.355278 26565 consensus_queue.cc:237] T 00000000000000000000000000000000 P ed06f54ab7ea4463a0edc5b64e7fd41d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed06f54ab7ea4463a0edc5b64e7fd41d" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 39729 } }
I20260806 06:26:40.355667 26568 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed06f54ab7ea4463a0edc5b64e7fd41d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ed06f54ab7ea4463a0edc5b64e7fd41d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed06f54ab7ea4463a0edc5b64e7fd41d" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 39729 } } }
I20260806 06:26:40.355742 26568 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed06f54ab7ea4463a0edc5b64e7fd41d [sys.catalog]: This master's current role is: LEADER
I20260806 06:26:40.355906 26571 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed06f54ab7ea4463a0edc5b64e7fd41d [sys.catalog]: SysCatalogTable state changed. Reason: New leader ed06f54ab7ea4463a0edc5b64e7fd41d. Latest consensus state: current_term: 1 leader_uuid: "ed06f54ab7ea4463a0edc5b64e7fd41d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed06f54ab7ea4463a0edc5b64e7fd41d" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 39729 } } }
I20260806 06:26:40.355966 26571 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed06f54ab7ea4463a0edc5b64e7fd41d [sys.catalog]: This master's current role is: LEADER
I20260806 06:26:40.356285 26580 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 06:26:40.356825 26580 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 06:26:40.357789 26562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:40.346766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50734 (local address 127.21.12.190:39729)
0806 06:26:40.346961 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0806 06:26:40.346970 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 06:26:40.347013 (+ 43us) server_negotiation.cc:407] Connection header received
0806 06:26:40.347618 (+ 605us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:26:40.347628 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:26:40.347693 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:26:40.347842 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:26:40.348486 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:40.349811 (+ 1325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:40.350656 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:40.351153 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:40.351263 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:40.353861 (+ 2598us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:26:40.353866 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:26:40.353866 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 06:26:40.353879 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:26:40.355839 (+ 1960us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:26:40.356288 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:26:40.356291 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:26:40.356292 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:26:40.356424 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:26:40.356677 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:26:40.356678 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:26:40.356678 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:26:40.356974 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:26:40.357134 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0806 06:26:40.357453 (+ 319us) server_negotiation.cc:299] Negotiation successful
0806 06:26:40.357604 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":95,"threads_started":1}
I20260806 06:26:40.358418 26580 catalog_manager.cc:1347] Generated new cluster ID: 067d97ad067b4cf5be8134f27391e918
I20260806 06:26:40.358469 26580 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 06:26:40.368522 26580 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 06:26:40.369005 26580 catalog_manager.cc:1504] Loading token signing keys...
I20260806 06:26:40.389760 26580 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ed06f54ab7ea4463a0edc5b64e7fd41d: Generated new TSK 0
I20260806 06:26:40.389933 26580 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.12.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.12.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.12.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.129.keytab.
Entry for principal kudu/127.21.12.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.129.keytab.
I20260806 06:26:40.424860 21554 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHhLhT/build/release/bin/kudu
/tmp/dist-test-taskFHhLhT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785997581049705-21554-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785997581049705-21554-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785997581049705-21554-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785997581049705-21554-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.12.129:0
--local_ip_for_outbound_sockets=127.21.12.129
--webserver_interface=127.21.12.129
--webserver_port=0
--tserver_master_addrs=127.21.12.190:39729
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.129.keytab
--principal=kudu/127.21.12.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.148:45671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785997581049705-21554-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785997581049705-21554-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785997581049705-21554-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 06:26:40.503751 26601 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 06:26:40.503939 26601 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 06:26:40.503968 26601 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 06:26:40.505143 26601 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 06:26:40.505183 26601 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 06:26:40.505231 26601 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.12.129
I20260806 06:26:40.506739 26601 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.148:45671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785997581049705-21554-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785997581049705-21554-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.129.keytab
--principal=kudu/127.21.12.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.12.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785997581049705-21554-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.12.129
--webserver_port=0
--tserver_master_addrs=127.21.12.190:39729
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26601
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.12.129
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785997581049705-21554-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 05:11:02 UTC on ccycloud.quasar-hvzbbz.root.comops.site
build id 4569617
I20260806 06:26:40.506951 26601 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 06:26:40.507150 26601 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 06:26:40.511874 26612 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:26:40.511914 26609 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:26:40.511904 26610 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:26:40.512079 26601 server_base.cc:1034] running on GCE node
I20260806 06:26:40.512281 26601 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 06:26:40.512496 26601 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 06:26:40.513641 26601 hybrid_clock.cc:648] HybridClock initialized: now 1785997600513620 us; error 31 us; skew 500 ppm
Aug 06 06:26:40 dist-test-slave-4tdj krb5kdc[26422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997600, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 06:26:40.516049 26601 init.cc:377] Logged in from keytab as kudu/127.21.12.129@KRBTEST.COM (short username kudu)
I20260806 06:26:40.516451 26601 webserver.cc:460] Webserver started at http://127.21.12.129:32865/ using document root <none> and password file <none>
I20260806 06:26:40.516659 26601 fs_manager.cc:359] Metadata directory not provided
I20260806 06:26:40.516705 26601 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 06:26:40.516824 26601 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 06:26:40.517808 26601 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785997581049705-21554-0/minicluster-data/ts-0/data/instance:
uuid: "e426b1278b93408bba76d6d77f599a61"
format_stamp: "Formatted at 2026-08-06 06:26:40 on dist-test-slave-4tdj"
server_key: "865ffcac6894f07e5696b7fc9104c092"
server_key_iv: "58c700fd2d09db2ab031f70fa2bb87ad"
server_key_version: "encryptionkey@0"
I20260806 06:26:40.518126 26601 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785997581049705-21554-0/minicluster-data/ts-0/wal/instance:
uuid: "e426b1278b93408bba76d6d77f599a61"
format_stamp: "Formatted at 2026-08-06 06:26:40 on dist-test-slave-4tdj"
server_key: "865ffcac6894f07e5696b7fc9104c092"
server_key_iv: "58c700fd2d09db2ab031f70fa2bb87ad"
server_key_version: "encryptionkey@0"
I20260806 06:26:40.519357 26601 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260806 06:26:40.520107 26625 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 06:26:40.520305 26601 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260806 06:26:40.520386 26601 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785997581049705-21554-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785997581049705-21554-0/minicluster-data/ts-0/wal
uuid: "e426b1278b93408bba76d6d77f599a61"
format_stamp: "Formatted at 2026-08-06 06:26:40 on dist-test-slave-4tdj"
server_key: "865ffcac6894f07e5696b7fc9104c092"
server_key_iv: "58c700fd2d09db2ab031f70fa2bb87ad"
server_key_version: "encryptionkey@0"
I20260806 06:26:40.520445 26601 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785997581049705-21554-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785997581049705-21554-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785997581049705-21554-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 06:26:40.543943 26601 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 06:26:40.546031 26601 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 06:26:40.546164 26601 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 06:26:40.546399 26601 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 06:26:40.546686 26601 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 06:26:40.546721 26601 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 06:26:40.546749 26601 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 06:26:40.546772 26601 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 06:26:40.552237 26601 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.129:40089
I20260806 06:26:40.552300 26797 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.129:40089 every 8 connection(s)
I20260806 06:26:40.552577 26601 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785997581049705-21554-0/minicluster-data/ts-0/data/info.pb
Aug 06 06:26:40 dist-test-slave-4tdj krb5kdc[26422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997600, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.129@KRBTEST.COM for kudu/127.21.12.190@KRBTEST.COM
I20260806 06:26:40.560092 21554 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHhLhT/build/release/bin/kudu as pid 26601
I20260806 06:26:40.560175 21554 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785997581049705-21554-0/minicluster-data/ts-0/wal/instance
I20260806 06:26:40.560375 21554 external_mini_cluster.cc:1428] Setting key ac75d68642beda547cbc9dd6bb2eeab8
I20260806 06:26:40.561596 26562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:40.553272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.129:36783 (local address 127.21.12.190:39729)
0806 06:26:40.553315 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0806 06:26:40.553316 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0806 06:26:40.553529 (+ 213us) server_negotiation.cc:407] Connection header received
0806 06:26:40.554459 (+ 930us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:26:40.554461 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:26:40.554473 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:26:40.554520 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:26:40.555164 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:40.555693 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:40.556654 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:40.556861 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:40.556915 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:40.558957 (+ 2042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:26:40.558958 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:26:40.558958 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 06:26:40.558965 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:26:40.560134 (+ 1169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:26:40.560697 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:26:40.560698 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:26:40.560698 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:26:40.560755 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:26:40.560988 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:26:40.560989 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:26:40.560989 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:26:40.561123 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:26:40.561196 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0806 06:26:40.561423 (+ 227us) server_negotiation.cc:299] Negotiation successful
0806 06:26:40.561511 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260806 06:26:40.561662 26801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:40.553254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:39729 (local address 127.21.12.129:36783)
0806 06:26:40.553408 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0806 06:26:40.553441 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0806 06:26:40.554346 (+ 905us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:26:40.554656 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:26:40.554661 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:26:40.554893 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0806 06:26:40.555074 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:40.555078 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:40.555772 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:40.555774 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:40.556575 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:40.556577 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:40.556928 (+ 351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:40.556929 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:40.557046 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:40.557460 (+ 414us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 06:26:40.557468 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0806 06:26:40.558863 (+ 1395us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 06:26:40.560234 (+ 1371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:26:40.560242 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:26:40.560244 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:26:40.560583 (+ 339us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:26:40.560818 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:26:40.560819 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:26:40.560819 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:26:40.560926 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:26:40.561186 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 06:26:40.561191 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 06:26:40.561303 (+ 112us) client_negotiation.cc:769] Sending connection context
0806 06:26:40.561384 (+ 81us) client_negotiation.cc:240] Negotiation successful
0806 06:26:40.561499 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":77,"threads_started":1}
I20260806 06:26:40.562106 26799 heartbeater.cc:347] Connected to a master server at 127.21.12.190:39729
I20260806 06:26:40.562193 26799 heartbeater.cc:464] Registering TS with master...
I20260806 06:26:40.562390 26799 heartbeater.cc:511] Master 127.21.12.190:39729 requested a full tablet report, sending...
I20260806 06:26:40.562913 26494 ts_manager.cc:194] Registered new tserver with Master: e426b1278b93408bba76d6d77f599a61 (127.21.12.129:40089)
I20260806 06:26:40.563493 26494 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.12.129@KRBTEST.COM'} at 127.21.12.129:36783
WARNING: no policy specified for kudu/127.21.12.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.12.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.12.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.130.keytab.
Entry for principal kudu/127.21.12.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.130.keytab.
I20260806 06:26:40.585305 21554 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHhLhT/build/release/bin/kudu
/tmp/dist-test-taskFHhLhT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785997581049705-21554-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785997581049705-21554-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785997581049705-21554-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785997581049705-21554-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.12.130:0
--local_ip_for_outbound_sockets=127.21.12.130
--webserver_interface=127.21.12.130
--webserver_port=0
--tserver_master_addrs=127.21.12.190:39729
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.130.keytab
--principal=kudu/127.21.12.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.148:45671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785997581049705-21554-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785997581049705-21554-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785997581049705-21554-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 06:26:40.664551 26806 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 06:26:40.664747 26806 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 06:26:40.664780 26806 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 06:26:40.665918 26806 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 06:26:40.665959 26806 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 06:26:40.665993 26806 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.12.130
I20260806 06:26:40.667538 26806 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.148:45671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785997581049705-21554-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785997581049705-21554-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.130.keytab
--principal=kudu/127.21.12.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.12.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785997581049705-21554-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.12.130
--webserver_port=0
--tserver_master_addrs=127.21.12.190:39729
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26806
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.12.130
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785997581049705-21554-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 05:11:02 UTC on ccycloud.quasar-hvzbbz.root.comops.site
build id 4569617
I20260806 06:26:40.667773 26806 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 06:26:40.667979 26806 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 06:26:40.672761 26813 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:26:40.672781 26815 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:26:40.672922 26806 server_base.cc:1034] running on GCE node
W20260806 06:26:40.672780 26812 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:26:40.673184 26806 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 06:26:40.673401 26806 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 06:26:40.674539 26806 hybrid_clock.cc:648] HybridClock initialized: now 1785997600674526 us; error 29 us; skew 500 ppm
Aug 06 06:26:40 dist-test-slave-4tdj krb5kdc[26422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997600, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 06:26:40.676854 26806 init.cc:377] Logged in from keytab as kudu/127.21.12.130@KRBTEST.COM (short username kudu)
I20260806 06:26:40.677253 26806 webserver.cc:460] Webserver started at http://127.21.12.130:38717/ using document root <none> and password file <none>
I20260806 06:26:40.677451 26806 fs_manager.cc:359] Metadata directory not provided
I20260806 06:26:40.677498 26806 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 06:26:40.677620 26806 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 06:26:40.678642 26806 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785997581049705-21554-0/minicluster-data/ts-1/data/instance:
uuid: "40981aca3d844b03b4f490d551240c4c"
format_stamp: "Formatted at 2026-08-06 06:26:40 on dist-test-slave-4tdj"
server_key: "2681cfb561f96220ab5cfa117f6e2738"
server_key_iv: "f9a48092ff22730a87840fe4f78d859d"
server_key_version: "encryptionkey@0"
I20260806 06:26:40.678964 26806 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785997581049705-21554-0/minicluster-data/ts-1/wal/instance:
uuid: "40981aca3d844b03b4f490d551240c4c"
format_stamp: "Formatted at 2026-08-06 06:26:40 on dist-test-slave-4tdj"
server_key: "2681cfb561f96220ab5cfa117f6e2738"
server_key_iv: "f9a48092ff22730a87840fe4f78d859d"
server_key_version: "encryptionkey@0"
I20260806 06:26:40.680219 26806 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260806 06:26:40.680851 26828 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 06:26:40.681000 26806 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260806 06:26:40.681084 26806 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785997581049705-21554-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785997581049705-21554-0/minicluster-data/ts-1/wal
uuid: "40981aca3d844b03b4f490d551240c4c"
format_stamp: "Formatted at 2026-08-06 06:26:40 on dist-test-slave-4tdj"
server_key: "2681cfb561f96220ab5cfa117f6e2738"
server_key_iv: "f9a48092ff22730a87840fe4f78d859d"
server_key_version: "encryptionkey@0"
I20260806 06:26:40.681140 26806 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785997581049705-21554-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785997581049705-21554-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785997581049705-21554-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 06:26:40.696195 26806 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 06:26:40.698052 26806 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 06:26:40.698179 26806 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 06:26:40.698441 26806 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 06:26:40.698738 26806 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 06:26:40.698771 26806 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 06:26:40.698797 26806 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 06:26:40.698823 26806 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 06:26:40.704268 26806 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.130:33047
I20260806 06:26:40.704326 27000 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.130:33047 every 8 connection(s)
I20260806 06:26:40.704610 26806 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785997581049705-21554-0/minicluster-data/ts-1/data/info.pb
I20260806 06:26:40.709964 21554 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHhLhT/build/release/bin/kudu as pid 26806
I20260806 06:26:40.710050 21554 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785997581049705-21554-0/minicluster-data/ts-1/wal/instance
I20260806 06:26:40.710212 21554 external_mini_cluster.cc:1428] Setting key 0cabe59f4bd3480a8176d03b55440d12
Aug 06 06:26:40 dist-test-slave-4tdj krb5kdc[26422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997600, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.130@KRBTEST.COM for kudu/127.21.12.190@KRBTEST.COM
I20260806 06:26:40.713786 26562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:40.705406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.130:45485 (local address 127.21.12.190:39729)
0806 06:26:40.705451 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0806 06:26:40.705453 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:26:40.705651 (+ 198us) server_negotiation.cc:407] Connection header received
0806 06:26:40.706613 (+ 962us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:26:40.706614 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:26:40.706624 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:26:40.706667 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:26:40.707248 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:40.707728 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:40.708695 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:40.708878 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:40.708925 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:40.711114 (+ 2189us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:26:40.711115 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:26:40.711115 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 06:26:40.711123 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:26:40.712464 (+ 1341us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:26:40.712940 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:26:40.712941 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:26:40.712941 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:26:40.712992 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:26:40.713237 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:26:40.713237 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:26:40.713237 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:26:40.713329 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:26:40.713372 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0806 06:26:40.713656 (+ 284us) server_negotiation.cc:299] Negotiation successful
0806 06:26:40.713722 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260806 06:26:40.714015 27003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:40.705368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:39729 (local address 127.21.12.130:45485)
0806 06:26:40.705526 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0806 06:26:40.705569 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0806 06:26:40.706506 (+ 937us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:26:40.706738 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:26:40.706741 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:26:40.706987 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0806 06:26:40.707183 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:40.707187 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:40.707786 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:40.707787 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:40.708614 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:40.708619 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:40.708984 (+ 365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:40.708986 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:40.709098 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:40.709591 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 06:26:40.709599 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0806 06:26:40.711028 (+ 1429us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 06:26:40.712568 (+ 1540us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:26:40.712576 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:26:40.712578 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:26:40.712868 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:26:40.713053 (+ 185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:26:40.713054 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:26:40.713054 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:26:40.713183 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:26:40.713375 (+ 192us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 06:26:40.713380 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 06:26:40.713529 (+ 149us) client_negotiation.cc:769] Sending connection context
0806 06:26:40.713642 (+ 113us) client_negotiation.cc:240] Negotiation successful
0806 06:26:40.713802 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":92,"threads_started":1}
I20260806 06:26:40.714475 27001 heartbeater.cc:347] Connected to a master server at 127.21.12.190:39729
I20260806 06:26:40.714574 27001 heartbeater.cc:464] Registering TS with master...
I20260806 06:26:40.714747 27001 heartbeater.cc:511] Master 127.21.12.190:39729 requested a full tablet report, sending...
I20260806 06:26:40.715107 26494 ts_manager.cc:194] Registered new tserver with Master: 40981aca3d844b03b4f490d551240c4c (127.21.12.130:33047)
I20260806 06:26:40.715498 26494 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.12.130@KRBTEST.COM'} at 127.21.12.130:45485
WARNING: no policy specified for kudu/127.21.12.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.12.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.12.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.131.keytab.
Entry for principal kudu/127.21.12.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.131.keytab.
I20260806 06:26:40.737038 21554 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHhLhT/build/release/bin/kudu
/tmp/dist-test-taskFHhLhT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785997581049705-21554-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785997581049705-21554-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785997581049705-21554-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785997581049705-21554-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.12.131:0
--local_ip_for_outbound_sockets=127.21.12.131
--webserver_interface=127.21.12.131
--webserver_port=0
--tserver_master_addrs=127.21.12.190:39729
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.131.keytab
--principal=kudu/127.21.12.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.148:45671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785997581049705-21554-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785997581049705-21554-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785997581049705-21554-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 06:26:40.817320 27006 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 06:26:40.817517 27006 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 06:26:40.817540 27006 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 06:26:40.818701 27006 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 06:26:40.818742 27006 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 06:26:40.818768 27006 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.12.131
I20260806 06:26:40.820323 27006 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.148:45671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785997581049705-21554-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785997581049705-21554-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.131.keytab
--principal=kudu/127.21.12.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.12.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785997581049705-21554-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.12.131
--webserver_port=0
--tserver_master_addrs=127.21.12.190:39729
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27006
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.12.131
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785997581049705-21554-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 05:11:02 UTC on ccycloud.quasar-hvzbbz.root.comops.site
build id 4569617
I20260806 06:26:40.820542 27006 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 06:26:40.820751 27006 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 06:26:40.825554 27015 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:26:40.825599 27014 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:26:40.825671 27006 server_base.cc:1034] running on GCE node
W20260806 06:26:40.825605 27018 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:26:40.826030 27006 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 06:26:40.826273 27006 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 06:26:40.827448 27006 hybrid_clock.cc:648] HybridClock initialized: now 1785997600827437 us; error 29 us; skew 500 ppm
Aug 06 06:26:40 dist-test-slave-4tdj krb5kdc[26422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997600, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 06:26:40.829836 27006 init.cc:377] Logged in from keytab as kudu/127.21.12.131@KRBTEST.COM (short username kudu)
I20260806 06:26:40.830286 27006 webserver.cc:460] Webserver started at http://127.21.12.131:45471/ using document root <none> and password file <none>
I20260806 06:26:40.830547 27006 fs_manager.cc:359] Metadata directory not provided
I20260806 06:26:40.830602 27006 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 06:26:40.830736 27006 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 06:26:40.831753 27006 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785997581049705-21554-0/minicluster-data/ts-2/data/instance:
uuid: "42c0e894d3be4feea1175fdd8a7dc9f0"
format_stamp: "Formatted at 2026-08-06 06:26:40 on dist-test-slave-4tdj"
server_key: "101005cd7f51b1ee57d1a8889b50fa52"
server_key_iv: "30c741cb3b1866f40a067a1ca678c77c"
server_key_version: "encryptionkey@0"
I20260806 06:26:40.832093 27006 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785997581049705-21554-0/minicluster-data/ts-2/wal/instance:
uuid: "42c0e894d3be4feea1175fdd8a7dc9f0"
format_stamp: "Formatted at 2026-08-06 06:26:40 on dist-test-slave-4tdj"
server_key: "101005cd7f51b1ee57d1a8889b50fa52"
server_key_iv: "30c741cb3b1866f40a067a1ca678c77c"
server_key_version: "encryptionkey@0"
I20260806 06:26:40.833308 27006 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260806 06:26:40.833979 27028 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 06:26:40.834158 27006 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260806 06:26:40.834232 27006 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785997581049705-21554-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785997581049705-21554-0/minicluster-data/ts-2/wal
uuid: "42c0e894d3be4feea1175fdd8a7dc9f0"
format_stamp: "Formatted at 2026-08-06 06:26:40 on dist-test-slave-4tdj"
server_key: "101005cd7f51b1ee57d1a8889b50fa52"
server_key_iv: "30c741cb3b1866f40a067a1ca678c77c"
server_key_version: "encryptionkey@0"
I20260806 06:26:40.834321 27006 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785997581049705-21554-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785997581049705-21554-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785997581049705-21554-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 06:26:40.846824 27006 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 06:26:40.848428 27006 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 06:26:40.848555 27006 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 06:26:40.848758 27006 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 06:26:40.849040 27006 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 06:26:40.849074 27006 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 06:26:40.849100 27006 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 06:26:40.849124 27006 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 06:26:40.854777 27006 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.131:41371
I20260806 06:26:40.854856 27188 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.131:41371 every 8 connection(s)
I20260806 06:26:40.855158 27006 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785997581049705-21554-0/minicluster-data/ts-2/data/info.pb
Aug 06 06:26:40 dist-test-slave-4tdj krb5kdc[26422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997600, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.131@KRBTEST.COM for kudu/127.21.12.190@KRBTEST.COM
I20260806 06:26:40.862011 21554 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHhLhT/build/release/bin/kudu as pid 27006
I20260806 06:26:40.862103 21554 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785997581049705-21554-0/minicluster-data/ts-2/wal/instance
I20260806 06:26:40.862334 21554 external_mini_cluster.cc:1428] Setting key 3a3a2fe7557b9bc47dfb82a2b17ad078
I20260806 06:26:40.864444 26562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:40.855891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.131:52363 (local address 127.21.12.190:39729)
0806 06:26:40.855939 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0806 06:26:40.855941 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:26:40.856210 (+ 269us) server_negotiation.cc:407] Connection header received
0806 06:26:40.857148 (+ 938us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:26:40.857149 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:26:40.857160 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:26:40.857216 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:26:40.857931 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:40.858445 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:40.859416 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:40.859645 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:40.859715 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:40.861686 (+ 1971us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:26:40.861688 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:26:40.861688 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 06:26:40.861694 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:26:40.863297 (+ 1603us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:26:40.863681 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:26:40.863682 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:26:40.863682 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:26:40.863735 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:26:40.863924 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:26:40.863924 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:26:40.863924 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:26:40.864016 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:26:40.864062 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0806 06:26:40.864323 (+ 261us) server_negotiation.cc:299] Negotiation successful
0806 06:26:40.864381 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260806 06:26:40.864637 27196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:40.855867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:39729 (local address 127.21.12.131:52363)
0806 06:26:40.856067 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0806 06:26:40.856112 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0806 06:26:40.857047 (+ 935us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:26:40.857269 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:26:40.857272 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:26:40.857588 (+ 316us) client_negotiation.cc:189] Negotiated authn=SASL
0806 06:26:40.857847 (+ 259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:40.857853 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:40.858526 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:40.858527 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:40.859316 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:40.859319 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:40.859726 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:40.859727 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:40.859829 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:40.860253 (+ 424us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 06:26:40.860261 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0806 06:26:40.861595 (+ 1334us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 06:26:40.863379 (+ 1784us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:26:40.863385 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:26:40.863386 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:26:40.863613 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:26:40.863783 (+ 170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:26:40.863784 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:26:40.863784 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:26:40.863878 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:26:40.864089 (+ 211us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 06:26:40.864096 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 06:26:40.864212 (+ 116us) client_negotiation.cc:769] Sending connection context
0806 06:26:40.864306 (+ 94us) client_negotiation.cc:240] Negotiation successful
0806 06:26:40.864425 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":103,"threads_started":1}
I20260806 06:26:40.865206 27189 heartbeater.cc:347] Connected to a master server at 127.21.12.190:39729
I20260806 06:26:40.865301 27189 heartbeater.cc:464] Registering TS with master...
I20260806 06:26:40.865471 27189 heartbeater.cc:511] Master 127.21.12.190:39729 requested a full tablet report, sending...
I20260806 06:26:40.865864 26494 ts_manager.cc:194] Registered new tserver with Master: 42c0e894d3be4feea1175fdd8a7dc9f0 (127.21.12.131:41371)
I20260806 06:26:40.866210 26494 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.12.131@KRBTEST.COM'} at 127.21.12.131:52363
I20260806 06:26:40.874145 21554 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 06 06:26:40 dist-test-slave-4tdj krb5kdc[26422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997600, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 06 06:26:40 dist-test-slave-4tdj krb5kdc[26422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997600, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.12.129@KRBTEST.COM
I20260806 06:26:40.897682 27203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:40.889938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48298 (local address 127.21.12.129:40089)
0806 06:26:40.890134 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0806 06:26:40.890141 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 06:26:40.890153 (+ 12us) server_negotiation.cc:407] Connection header received
0806 06:26:40.890182 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:26:40.890185 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:26:40.890202 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:26:40.890281 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:26:40.890927 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:40.891653 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:40.892408 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:40.892748 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:40.892810 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:40.895023 (+ 2213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:26:40.895038 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:26:40.895039 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 06:26:40.895046 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:26:40.896354 (+ 1308us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:26:40.896705 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:26:40.896709 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:26:40.896710 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:26:40.896774 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:26:40.896983 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:26:40.896984 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:26:40.896985 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:26:40.897166 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:26:40.897318 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0806 06:26:40.897444 (+ 126us) server_negotiation.cc:299] Negotiation successful
0806 06:26:40.897551 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":102,"threads_started":1}
Aug 06 06:26:40 dist-test-slave-4tdj krb5kdc[26422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997600, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.12.190@KRBTEST.COM
I20260806 06:26:40.905038 26562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:40.899008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50750 (local address 127.21.12.190:39729)
0806 06:26:40.899068 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0806 06:26:40.899070 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:26:40.899079 (+ 9us) server_negotiation.cc:407] Connection header received
0806 06:26:40.899139 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:26:40.899140 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:26:40.899150 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:26:40.899191 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:26:40.899582 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:40.900059 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:40.900691 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:40.900869 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:40.900922 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:40.902950 (+ 2028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:26:40.902951 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:26:40.902951 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 06:26:40.902957 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:26:40.904027 (+ 1070us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:26:40.904341 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:26:40.904342 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:26:40.904343 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:26:40.904409 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:26:40.904560 (+ 151us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:26:40.904560 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:26:40.904561 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:26:40.904699 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:26:40.904774 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0806 06:26:40.904862 (+ 88us) server_negotiation.cc:299] Negotiation successful
0806 06:26:40.904928 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
W20260806 06:26:40.905287 26494 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50750
I20260806 06:26:40.905629 21554 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHhLhT/build/release/bin/kudu with pid 26601
I20260806 06:26:40.914793 21554 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHhLhT/build/release/bin/kudu with pid 26806
I20260806 06:26:40.927464 21554 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHhLhT/build/release/bin/kudu with pid 27006
I20260806 06:26:40.938838 21554 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHhLhT/build/release/bin/kudu with pid 26439
2026-08-06T06:26:40Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (1966 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785997581049705-21554-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 06:26:40 dist-test-slave-4tdj krb5kdc[27209](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 06:26:40 dist-test-slave-4tdj krb5kdc[27209](info): set up 2 sockets
Aug 06 06:26:40 dist-test-slave-4tdj krb5kdc[27209](info): commencing operation
krb5kdc: starting...
W20260806 06:26:42.052235 21554 mini_kdc.cc:121] Time spent starting KDC: real 1.089s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785997581049705-21554-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785997581049705-21554-0/krb5kdc/test-user.keytab.
Aug 06 06:26:42 dist-test-slave-4tdj krb5kdc[27209](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997602, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T06:26:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T06:26:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.12.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.12.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.12.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.190.keytab.
Entry for principal kudu/127.21.12.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.190.keytab.
I20260806 06:26:42.165170 21554 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHhLhT/build/release/bin/kudu
/tmp/dist-test-taskFHhLhT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785997581049705-21554-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785997581049705-21554-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785997581049705-21554-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785997581049705-21554-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.12.190:40387
--webserver_interface=127.21.12.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.190.keytab
--principal=kudu/127.21.12.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.148:39579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.12.190:40387
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785997581049705-21554-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785997581049705-21554-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785997581049705-21554-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 06:26:42.243728 27229 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 06:26:42.243914 27229 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 06:26:42.243944 27229 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 06:26:42.245078 27229 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 06:26:42.245119 27229 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 06:26:42.245137 27229 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 06:26:42.245154 27229 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 06:26:42.245168 27229 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 06:26:42.246716 27229 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.148:39579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785997581049705-21554-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785997581049705-21554-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.12.190:40387
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.190.keytab
--principal=kudu/127.21.12.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.12.190:40387
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785997581049705-21554-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.12.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27229
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785997581049705-21554-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 05:11:02 UTC on ccycloud.quasar-hvzbbz.root.comops.site
build id 4569617
I20260806 06:26:42.246953 27229 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 06:26:42.247153 27229 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 06:26:42.251762 27240 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:26:42.251778 27242 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:26:42.251770 27238 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:26:42.252068 27229 server_base.cc:1034] running on GCE node
I20260806 06:26:42.252295 27229 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 06:26:42.252568 27229 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 06:26:42.253718 27229 hybrid_clock.cc:648] HybridClock initialized: now 1785997602253720 us; error 45 us; skew 500 ppm
Aug 06 06:26:42 dist-test-slave-4tdj krb5kdc[27209](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997602, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 06:26:42.256198 27229 init.cc:377] Logged in from keytab as kudu/127.21.12.190@KRBTEST.COM (short username kudu)
I20260806 06:26:42.256567 27229 fs_manager.cc:359] Metadata directory not provided
I20260806 06:26:42.256618 27229 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 06:26:42.256743 27229 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 06:26:42.257776 27229 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785997581049705-21554-0/minicluster-data/master-0/data/instance:
uuid: "1b404aadc7de4ea0b15dde51d60d11a1"
format_stamp: "Formatted at 2026-08-06 06:26:42 on dist-test-slave-4tdj"
server_key: "c8c0f27911895413d4f9dfbebe7eeaf1"
server_key_iv: "008f2d6d94af719a9ed663596bc1d9eb"
server_key_version: "encryptionkey@0"
I20260806 06:26:42.258100 27229 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785997581049705-21554-0/minicluster-data/master-0/wal/instance:
uuid: "1b404aadc7de4ea0b15dde51d60d11a1"
format_stamp: "Formatted at 2026-08-06 06:26:42 on dist-test-slave-4tdj"
server_key: "c8c0f27911895413d4f9dfbebe7eeaf1"
server_key_iv: "008f2d6d94af719a9ed663596bc1d9eb"
server_key_version: "encryptionkey@0"
I20260806 06:26:42.259375 27229 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260806 06:26:42.260149 27254 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 06:26:42.260303 27229 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260806 06:26:42.260378 27229 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785997581049705-21554-0/minicluster-data/master-0/data,/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785997581049705-21554-0/minicluster-data/master-0/wal
uuid: "1b404aadc7de4ea0b15dde51d60d11a1"
format_stamp: "Formatted at 2026-08-06 06:26:42 on dist-test-slave-4tdj"
server_key: "c8c0f27911895413d4f9dfbebe7eeaf1"
server_key_iv: "008f2d6d94af719a9ed663596bc1d9eb"
server_key_version: "encryptionkey@0"
I20260806 06:26:42.260442 27229 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785997581049705-21554-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785997581049705-21554-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785997581049705-21554-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 06:26:42.281404 27229 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 06:26:42.283229 27229 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 06:26:42.283360 27229 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 06:26:42.287122 27229 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.190:40387
I20260806 06:26:42.287155 27322 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.190:40387 every 8 connection(s)
I20260806 06:26:42.287416 27229 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785997581049705-21554-0/minicluster-data/master-0/data/info.pb
I20260806 06:26:42.288394 27323 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 06:26:42.289402 21554 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHhLhT/build/release/bin/kudu as pid 27229
I20260806 06:26:42.289518 21554 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785997581049705-21554-0/minicluster-data/master-0/wal/instance
I20260806 06:26:42.289710 21554 external_mini_cluster.cc:1428] Setting key e2ead8533ba37e39fed3f5949454c0db
I20260806 06:26:42.291059 27323 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b404aadc7de4ea0b15dde51d60d11a1: Bootstrap starting.
I20260806 06:26:42.291774 27323 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1b404aadc7de4ea0b15dde51d60d11a1: Neither blocks nor log segments found. Creating new log.
I20260806 06:26:42.292060 27323 log.cc:824] T 00000000000000000000000000000000 P 1b404aadc7de4ea0b15dde51d60d11a1: Log is configured to *not* fsync() on all Append() calls
I20260806 06:26:42.292706 27323 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b404aadc7de4ea0b15dde51d60d11a1: No bootstrap required, opened a new log
I20260806 06:26:42.294963 27323 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1b404aadc7de4ea0b15dde51d60d11a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b404aadc7de4ea0b15dde51d60d11a1" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 40387 } }
I20260806 06:26:42.295143 27323 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1b404aadc7de4ea0b15dde51d60d11a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 06:26:42.295181 27323 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1b404aadc7de4ea0b15dde51d60d11a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b404aadc7de4ea0b15dde51d60d11a1, State: Initialized, Role: FOLLOWER
I20260806 06:26:42.295323 27323 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1b404aadc7de4ea0b15dde51d60d11a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b404aadc7de4ea0b15dde51d60d11a1" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 40387 } }
I20260806 06:26:42.295408 27323 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1b404aadc7de4ea0b15dde51d60d11a1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 06:26:42.295439 27323 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1b404aadc7de4ea0b15dde51d60d11a1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 06:26:42.295480 27323 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1b404aadc7de4ea0b15dde51d60d11a1 [term 0 FOLLOWER]: Advancing to term 1
I20260806 06:26:42.296067 27323 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1b404aadc7de4ea0b15dde51d60d11a1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b404aadc7de4ea0b15dde51d60d11a1" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 40387 } }
I20260806 06:26:42.296154 27323 leader_election.cc:302] T 00000000000000000000000000000000 P 1b404aadc7de4ea0b15dde51d60d11a1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1b404aadc7de4ea0b15dde51d60d11a1; no voters:
I20260806 06:26:42.296295 27323 leader_election.cc:288] T 00000000000000000000000000000000 P 1b404aadc7de4ea0b15dde51d60d11a1 [CANDIDATE]: Term 1 election: Requested vote from peers
Aug 06 06:26:42 dist-test-slave-4tdj krb5kdc[27209](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997602, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.12.190@KRBTEST.COM
I20260806 06:26:42.296336 27329 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1b404aadc7de4ea0b15dde51d60d11a1 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 06:26:42.296446 27329 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1b404aadc7de4ea0b15dde51d60d11a1 [term 1 LEADER]: Becoming Leader. State: Replica: 1b404aadc7de4ea0b15dde51d60d11a1, State: Running, Role: LEADER
I20260806 06:26:42.296558 27323 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1b404aadc7de4ea0b15dde51d60d11a1 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 06:26:42.296600 27329 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1b404aadc7de4ea0b15dde51d60d11a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b404aadc7de4ea0b15dde51d60d11a1" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 40387 } }
I20260806 06:26:42.297142 27331 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b404aadc7de4ea0b15dde51d60d11a1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1b404aadc7de4ea0b15dde51d60d11a1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b404aadc7de4ea0b15dde51d60d11a1" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 40387 } } }
I20260806 06:26:42.297163 27330 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b404aadc7de4ea0b15dde51d60d11a1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1b404aadc7de4ea0b15dde51d60d11a1. Latest consensus state: current_term: 1 leader_uuid: "1b404aadc7de4ea0b15dde51d60d11a1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b404aadc7de4ea0b15dde51d60d11a1" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 40387 } } }
I20260806 06:26:42.297304 27331 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b404aadc7de4ea0b15dde51d60d11a1 [sys.catalog]: This master's current role is: LEADER
I20260806 06:26:42.297391 27330 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b404aadc7de4ea0b15dde51d60d11a1 [sys.catalog]: This master's current role is: LEADER
I20260806 06:26:42.297770 27340 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 06:26:42.298911 27340 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 06:26:42.300144 27327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:42.290210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46508 (local address 127.21.12.190:40387)
0806 06:26:42.290369 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0806 06:26:42.290379 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 06:26:42.290408 (+ 29us) server_negotiation.cc:407] Connection header received
0806 06:26:42.290857 (+ 449us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:26:42.290864 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:26:42.290910 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:26:42.291029 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:26:42.291755 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:42.292808 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:42.293854 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:42.294241 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:42.294348 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:42.296865 (+ 2517us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:26:42.296868 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:26:42.296868 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 06:26:42.296880 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:26:42.298694 (+ 1814us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:26:42.299010 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:26:42.299014 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:26:42.299014 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:26:42.299110 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:26:42.299282 (+ 172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:26:42.299283 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:26:42.299284 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:26:42.299526 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:26:42.299659 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0806 06:26:42.299816 (+ 157us) server_negotiation.cc:299] Negotiation successful
0806 06:26:42.299944 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":87,"threads_started":1}
I20260806 06:26:42.300737 27340 catalog_manager.cc:1347] Generated new cluster ID: 8c6ac1d097904a63b7fd854fde409f0e
I20260806 06:26:42.300787 27340 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 06:26:42.318385 27340 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 06:26:42.318969 27340 catalog_manager.cc:1504] Loading token signing keys...
I20260806 06:26:42.323418 27340 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1b404aadc7de4ea0b15dde51d60d11a1: Generated new TSK 0
I20260806 06:26:42.323619 27340 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.12.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.12.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.12.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.129.keytab.
Entry for principal kudu/127.21.12.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.129.keytab.
I20260806 06:26:42.354156 21554 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHhLhT/build/release/bin/kudu
/tmp/dist-test-taskFHhLhT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785997581049705-21554-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785997581049705-21554-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785997581049705-21554-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785997581049705-21554-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.12.129:0
--local_ip_for_outbound_sockets=127.21.12.129
--webserver_interface=127.21.12.129
--webserver_port=0
--tserver_master_addrs=127.21.12.190:40387
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.129.keytab
--principal=kudu/127.21.12.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.148:39579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785997581049705-21554-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785997581049705-21554-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785997581049705-21554-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 06:26:42.435066 27362 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 06:26:42.435249 27362 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 06:26:42.435272 27362 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 06:26:42.436378 27362 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 06:26:42.436419 27362 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 06:26:42.436443 27362 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.12.129
I20260806 06:26:42.437899 27362 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.148:39579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785997581049705-21554-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785997581049705-21554-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.129.keytab
--principal=kudu/127.21.12.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.12.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785997581049705-21554-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.12.129
--webserver_port=0
--tserver_master_addrs=127.21.12.190:40387
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27362
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.12.129
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785997581049705-21554-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 05:11:02 UTC on ccycloud.quasar-hvzbbz.root.comops.site
build id 4569617
I20260806 06:26:42.438109 27362 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 06:26:42.438344 27362 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 06:26:42.443006 27371 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:26:42.442994 27370 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:26:42.442993 27374 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:26:42.443256 27362 server_base.cc:1034] running on GCE node
I20260806 06:26:42.443452 27362 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 06:26:42.443689 27362 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 06:26:42.444857 27362 hybrid_clock.cc:648] HybridClock initialized: now 1785997602444834 us; error 36 us; skew 500 ppm
Aug 06 06:26:42 dist-test-slave-4tdj krb5kdc[27209](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997602, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 06:26:42.447327 27362 init.cc:377] Logged in from keytab as kudu/127.21.12.129@KRBTEST.COM (short username kudu)
I20260806 06:26:42.447652 27362 fs_manager.cc:359] Metadata directory not provided
I20260806 06:26:42.447702 27362 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 06:26:42.447826 27362 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 06:26:42.448908 27362 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785997581049705-21554-0/minicluster-data/ts-0/data/instance:
uuid: "14e8034244c14f6aa3081e79abab34dd"
format_stamp: "Formatted at 2026-08-06 06:26:42 on dist-test-slave-4tdj"
server_key: "f10c225a73eebf58036e0f622f173f9a"
server_key_iv: "6ddac345e5e80b66c2797b15a21d41c4"
server_key_version: "encryptionkey@0"
I20260806 06:26:42.449227 27362 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785997581049705-21554-0/minicluster-data/ts-0/wal/instance:
uuid: "14e8034244c14f6aa3081e79abab34dd"
format_stamp: "Formatted at 2026-08-06 06:26:42 on dist-test-slave-4tdj"
server_key: "f10c225a73eebf58036e0f622f173f9a"
server_key_iv: "6ddac345e5e80b66c2797b15a21d41c4"
server_key_version: "encryptionkey@0"
I20260806 06:26:42.450388 27362 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260806 06:26:42.451068 27383 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 06:26:42.451217 27362 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260806 06:26:42.451298 27362 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785997581049705-21554-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785997581049705-21554-0/minicluster-data/ts-0/wal
uuid: "14e8034244c14f6aa3081e79abab34dd"
format_stamp: "Formatted at 2026-08-06 06:26:42 on dist-test-slave-4tdj"
server_key: "f10c225a73eebf58036e0f622f173f9a"
server_key_iv: "6ddac345e5e80b66c2797b15a21d41c4"
server_key_version: "encryptionkey@0"
I20260806 06:26:42.451359 27362 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785997581049705-21554-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785997581049705-21554-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785997581049705-21554-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 06:26:42.460359 27362 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 06:26:42.461906 27362 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 06:26:42.462029 27362 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 06:26:42.462226 27362 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 06:26:42.462591 27362 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 06:26:42.462627 27362 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 06:26:42.462654 27362 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 06:26:42.462671 27362 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 06:26:42.468076 27362 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.129:40521
I20260806 06:26:42.468137 27558 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.129:40521 every 8 connection(s)
I20260806 06:26:42.468348 27362 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785997581049705-21554-0/minicluster-data/ts-0/data/info.pb
I20260806 06:26:42.468628 21554 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHhLhT/build/release/bin/kudu as pid 27362
I20260806 06:26:42.468731 21554 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785997581049705-21554-0/minicluster-data/ts-0/wal/instance
I20260806 06:26:42.468914 21554 external_mini_cluster.cc:1428] Setting key db26087059c4957229442548053d15b0
Aug 06 06:26:42 dist-test-slave-4tdj krb5kdc[27209](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997602, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.129@KRBTEST.COM for kudu/127.21.12.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.21.12.130@KRBTEST.COM; defaulting to no policy
I20260806 06:26:42.477984 27327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:42.469094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.129:41077 (local address 127.21.12.190:40387)
0806 06:26:42.469145 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0806 06:26:42.469147 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:26:42.469346 (+ 199us) server_negotiation.cc:407] Connection header received
0806 06:26:42.469933 (+ 587us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:26:42.469934 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:26:42.469943 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:26:42.469996 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:26:42.471193 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:42.471922 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:42.472912 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:42.473121 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:42.473176 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:42.475446 (+ 2270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:26:42.475448 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:26:42.475448 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 06:26:42.475455 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:26:42.476658 (+ 1203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:26:42.477101 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:26:42.477102 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:26:42.477102 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:26:42.477146 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:26:42.477391 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:26:42.477391 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:26:42.477392 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:26:42.477483 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:26:42.477525 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0806 06:26:42.477847 (+ 322us) server_negotiation.cc:299] Negotiation successful
0806 06:26:42.477921 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260806 06:26:42.478206 27561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:42.469032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:40387 (local address 127.21.12.129:41077)
0806 06:26:42.469231 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0806 06:26:42.469274 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0806 06:26:42.469835 (+ 561us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:26:42.470069 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:26:42.470073 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:26:42.470432 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0806 06:26:42.471065 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:42.471072 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:42.472048 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:42.472049 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:42.472826 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:42.472829 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:42.473191 (+ 362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:42.473192 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:42.473319 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:42.473780 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 06:26:42.473791 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0806 06:26:42.475356 (+ 1565us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 06:26:42.476737 (+ 1381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:26:42.476745 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:26:42.476748 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:26:42.477031 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:26:42.477205 (+ 174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:26:42.477205 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:26:42.477206 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:26:42.477332 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:26:42.477544 (+ 212us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 06:26:42.477551 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 06:26:42.477706 (+ 155us) client_negotiation.cc:769] Sending connection context
0806 06:26:42.477822 (+ 116us) client_negotiation.cc:240] Negotiation successful
0806 06:26:42.477978 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":98,"threads_started":1}
I20260806 06:26:42.478688 27559 heartbeater.cc:347] Connected to a master server at 127.21.12.190:40387
I20260806 06:26:42.478770 27559 heartbeater.cc:464] Registering TS with master...
I20260806 06:26:42.478906 27559 heartbeater.cc:511] Master 127.21.12.190:40387 requested a full tablet report, sending...
I20260806 06:26:42.479389 27276 ts_manager.cc:194] Registered new tserver with Master: 14e8034244c14f6aa3081e79abab34dd (127.21.12.129:40521)
I20260806 06:26:42.479943 27276 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.12.129@KRBTEST.COM'} at 127.21.12.129:41077
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.12.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.12.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.130.keytab.
Entry for principal kudu/127.21.12.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.130.keytab.
I20260806 06:26:42.494508 21554 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHhLhT/build/release/bin/kudu
/tmp/dist-test-taskFHhLhT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785997581049705-21554-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785997581049705-21554-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785997581049705-21554-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785997581049705-21554-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.12.130:0
--local_ip_for_outbound_sockets=127.21.12.130
--webserver_interface=127.21.12.130
--webserver_port=0
--tserver_master_addrs=127.21.12.190:40387
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.130.keytab
--principal=kudu/127.21.12.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.148:39579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785997581049705-21554-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785997581049705-21554-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785997581049705-21554-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 06:26:42.573567 27567 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 06:26:42.573746 27567 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 06:26:42.573767 27567 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 06:26:42.574901 27567 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 06:26:42.574939 27567 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 06:26:42.574966 27567 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.12.130
I20260806 06:26:42.576429 27567 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.148:39579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785997581049705-21554-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785997581049705-21554-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.130.keytab
--principal=kudu/127.21.12.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.12.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785997581049705-21554-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.12.130
--webserver_port=0
--tserver_master_addrs=127.21.12.190:40387
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27567
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.12.130
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785997581049705-21554-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 05:11:02 UTC on ccycloud.quasar-hvzbbz.root.comops.site
build id 4569617
I20260806 06:26:42.576627 27567 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 06:26:42.576825 27567 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 06:26:42.581554 27572 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:26:42.581574 27573 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:26:42.581756 27575 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:26:42.581754 27567 server_base.cc:1034] running on GCE node
I20260806 06:26:42.581974 27567 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 06:26:42.582213 27567 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 06:26:42.583349 27567 hybrid_clock.cc:648] HybridClock initialized: now 1785997602583349 us; error 36 us; skew 500 ppm
Aug 06 06:26:42 dist-test-slave-4tdj krb5kdc[27209](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997602, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 06:26:42.585698 27567 init.cc:377] Logged in from keytab as kudu/127.21.12.130@KRBTEST.COM (short username kudu)
I20260806 06:26:42.585969 27567 fs_manager.cc:359] Metadata directory not provided
I20260806 06:26:42.586009 27567 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 06:26:42.586109 27567 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 06:26:42.587096 27567 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785997581049705-21554-0/minicluster-data/ts-1/data/instance:
uuid: "dd8bfd8a67284cee84b9f9074c16404e"
format_stamp: "Formatted at 2026-08-06 06:26:42 on dist-test-slave-4tdj"
server_key: "9503c10432cc9daade661a3cebdfd46a"
server_key_iv: "9b582e69deb4af4e9c4171bb322b5610"
server_key_version: "encryptionkey@0"
I20260806 06:26:42.587386 27567 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785997581049705-21554-0/minicluster-data/ts-1/wal/instance:
uuid: "dd8bfd8a67284cee84b9f9074c16404e"
format_stamp: "Formatted at 2026-08-06 06:26:42 on dist-test-slave-4tdj"
server_key: "9503c10432cc9daade661a3cebdfd46a"
server_key_iv: "9b582e69deb4af4e9c4171bb322b5610"
server_key_version: "encryptionkey@0"
I20260806 06:26:42.588486 27567 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260806 06:26:42.589120 27584 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 06:26:42.589260 27567 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260806 06:26:42.589383 27567 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785997581049705-21554-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785997581049705-21554-0/minicluster-data/ts-1/wal
uuid: "dd8bfd8a67284cee84b9f9074c16404e"
format_stamp: "Formatted at 2026-08-06 06:26:42 on dist-test-slave-4tdj"
server_key: "9503c10432cc9daade661a3cebdfd46a"
server_key_iv: "9b582e69deb4af4e9c4171bb322b5610"
server_key_version: "encryptionkey@0"
I20260806 06:26:42.589444 27567 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785997581049705-21554-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785997581049705-21554-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785997581049705-21554-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 06:26:42.618938 27567 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 06:26:42.620733 27567 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 06:26:42.620841 27567 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 06:26:42.621042 27567 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 06:26:42.621340 27567 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 06:26:42.621373 27567 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 06:26:42.621402 27567 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 06:26:42.621425 27567 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 06:26:42.626818 27567 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.130:42297
I20260806 06:26:42.626855 27744 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.130:42297 every 8 connection(s)
I20260806 06:26:42.627115 27567 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785997581049705-21554-0/minicluster-data/ts-1/data/info.pb
I20260806 06:26:42.629325 21554 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHhLhT/build/release/bin/kudu as pid 27567
I20260806 06:26:42.629424 21554 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785997581049705-21554-0/minicluster-data/ts-1/wal/instance
I20260806 06:26:42.629626 21554 external_mini_cluster.cc:1428] Setting key bf29eb2e18e6b780f44c3016c1f5fe40
Aug 06 06:26:42 dist-test-slave-4tdj krb5kdc[27209](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997602, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.130@KRBTEST.COM for kudu/127.21.12.190@KRBTEST.COM
I20260806 06:26:42.636133 27327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:42.627866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.130:50895 (local address 127.21.12.190:40387)
0806 06:26:42.627927 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0806 06:26:42.627929 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:26:42.628122 (+ 193us) server_negotiation.cc:407] Connection header received
0806 06:26:42.628573 (+ 451us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:26:42.628574 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:26:42.628583 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:26:42.628633 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:26:42.629522 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:42.630180 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:42.631183 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:42.631374 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:42.631430 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:42.633679 (+ 2249us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:26:42.633680 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:26:42.633680 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 06:26:42.633687 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:26:42.634931 (+ 1244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:26:42.635287 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:26:42.635288 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:26:42.635289 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:26:42.635341 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:26:42.635554 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:26:42.635555 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:26:42.635555 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:26:42.635656 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:26:42.635710 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0806 06:26:42.635956 (+ 246us) server_negotiation.cc:299] Negotiation successful
0806 06:26:42.636045 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260806 06:26:42.636274 27748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:42.627823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:40387 (local address 127.21.12.130:50895)
0806 06:26:42.628000 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0806 06:26:42.628049 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0806 06:26:42.628499 (+ 450us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:26:42.628695 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:26:42.628698 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:26:42.628938 (+ 240us) client_negotiation.cc:189] Negotiated authn=SASL
0806 06:26:42.629437 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:42.629444 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:42.630272 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:42.630274 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:42.631095 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:42.631098 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:42.631442 (+ 344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:42.631443 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:42.631558 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:42.632028 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 06:26:42.632038 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0806 06:26:42.633590 (+ 1552us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 06:26:42.635006 (+ 1416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:26:42.635014 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:26:42.635016 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:26:42.635221 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:26:42.635394 (+ 173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:26:42.635394 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:26:42.635394 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:26:42.635500 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:26:42.635709 (+ 209us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 06:26:42.635716 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 06:26:42.635842 (+ 126us) client_negotiation.cc:769] Sending connection context
0806 06:26:42.635927 (+ 85us) client_negotiation.cc:240] Negotiation successful
0806 06:26:42.636071 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":94,"threads_started":1}
I20260806 06:26:42.636683 27745 heartbeater.cc:347] Connected to a master server at 127.21.12.190:40387
I20260806 06:26:42.636770 27745 heartbeater.cc:464] Registering TS with master...
I20260806 06:26:42.636904 27745 heartbeater.cc:511] Master 127.21.12.190:40387 requested a full tablet report, sending...
I20260806 06:26:42.637267 27276 ts_manager.cc:194] Registered new tserver with Master: dd8bfd8a67284cee84b9f9074c16404e (127.21.12.130:42297)
I20260806 06:26:42.637773 27276 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.12.130@KRBTEST.COM'} at 127.21.12.130:50895
WARNING: no policy specified for kudu/127.21.12.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.12.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.12.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.131.keytab.
Entry for principal kudu/127.21.12.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.131.keytab.
I20260806 06:26:42.654170 21554 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHhLhT/build/release/bin/kudu
/tmp/dist-test-taskFHhLhT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785997581049705-21554-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785997581049705-21554-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785997581049705-21554-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785997581049705-21554-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.12.131:0
--local_ip_for_outbound_sockets=127.21.12.131
--webserver_interface=127.21.12.131
--webserver_port=0
--tserver_master_addrs=127.21.12.190:40387
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.131.keytab
--principal=kudu/127.21.12.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.148:39579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785997581049705-21554-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785997581049705-21554-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785997581049705-21554-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 06:26:42.734162 27756 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 06:26:42.734393 27756 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 06:26:42.734421 27756 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 06:26:42.735561 27756 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 06:26:42.735598 27756 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 06:26:42.735622 27756 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.12.131
I20260806 06:26:42.737114 27756 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.148:39579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785997581049705-21554-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785997581049705-21554-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.131.keytab
--principal=kudu/127.21.12.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.12.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785997581049705-21554-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.12.131
--webserver_port=0
--tserver_master_addrs=127.21.12.190:40387
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27756
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.12.131
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785997581049705-21554-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 05:11:02 UTC on ccycloud.quasar-hvzbbz.root.comops.site
build id 4569617
I20260806 06:26:42.737334 27756 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 06:26:42.737535 27756 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 06:26:42.742345 27763 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:26:42.742312 27771 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:26:42.742331 27768 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:26:42.742537 27756 server_base.cc:1034] running on GCE node
I20260806 06:26:42.742725 27756 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 06:26:42.742934 27756 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 06:26:42.744103 27756 hybrid_clock.cc:648] HybridClock initialized: now 1785997602744085 us; error 29 us; skew 500 ppm
Aug 06 06:26:42 dist-test-slave-4tdj krb5kdc[27209](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997602, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 06:26:42.746560 27756 init.cc:377] Logged in from keytab as kudu/127.21.12.131@KRBTEST.COM (short username kudu)
I20260806 06:26:42.746865 27756 fs_manager.cc:359] Metadata directory not provided
I20260806 06:26:42.746914 27756 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 06:26:42.747040 27756 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 06:26:42.748011 27756 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785997581049705-21554-0/minicluster-data/ts-2/data/instance:
uuid: "36cf9124d6a74980ae20eec5a5995758"
format_stamp: "Formatted at 2026-08-06 06:26:42 on dist-test-slave-4tdj"
server_key: "6eb000179766d4c5b9993f8bb40b742e"
server_key_iv: "73788384e582f697d06c680292989a8b"
server_key_version: "encryptionkey@0"
I20260806 06:26:42.748327 27756 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785997581049705-21554-0/minicluster-data/ts-2/wal/instance:
uuid: "36cf9124d6a74980ae20eec5a5995758"
format_stamp: "Formatted at 2026-08-06 06:26:42 on dist-test-slave-4tdj"
server_key: "6eb000179766d4c5b9993f8bb40b742e"
server_key_iv: "73788384e582f697d06c680292989a8b"
server_key_version: "encryptionkey@0"
I20260806 06:26:42.749478 27756 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260806 06:26:42.750102 27781 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 06:26:42.750267 27756 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260806 06:26:42.750371 27756 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785997581049705-21554-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785997581049705-21554-0/minicluster-data/ts-2/wal
uuid: "36cf9124d6a74980ae20eec5a5995758"
format_stamp: "Formatted at 2026-08-06 06:26:42 on dist-test-slave-4tdj"
server_key: "6eb000179766d4c5b9993f8bb40b742e"
server_key_iv: "73788384e582f697d06c680292989a8b"
server_key_version: "encryptionkey@0"
I20260806 06:26:42.750427 27756 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785997581049705-21554-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785997581049705-21554-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785997581049705-21554-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 06:26:42.779385 27756 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 06:26:42.781219 27756 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 06:26:42.781354 27756 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 06:26:42.781579 27756 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 06:26:42.781889 27756 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 06:26:42.781934 27756 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 06:26:42.781965 27756 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 06:26:42.781983 27756 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 06:26:42.787864 27756 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.131:37959
I20260806 06:26:42.787921 27962 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.131:37959 every 8 connection(s)
I20260806 06:26:42.788156 27756 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785997581049705-21554-0/minicluster-data/ts-2/data/info.pb
I20260806 06:26:42.789186 21554 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHhLhT/build/release/bin/kudu as pid 27756
I20260806 06:26:42.789286 21554 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785997581049705-21554-0/minicluster-data/ts-2/wal/instance
I20260806 06:26:42.789461 21554 external_mini_cluster.cc:1428] Setting key 449a2a3dbd4cfeef93b315a19e215e04
Aug 06 06:26:42 dist-test-slave-4tdj krb5kdc[27209](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997602, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.131@KRBTEST.COM for kudu/127.21.12.190@KRBTEST.COM
I20260806 06:26:42.796641 27327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:42.788804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.131:44165 (local address 127.21.12.190:40387)
0806 06:26:42.788855 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0806 06:26:42.788857 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:26:42.789038 (+ 181us) server_negotiation.cc:407] Connection header received
0806 06:26:42.789591 (+ 553us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:26:42.789593 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:26:42.789606 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:26:42.789652 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:26:42.790714 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:42.791212 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:42.792235 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:42.792435 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:42.792490 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:42.794513 (+ 2023us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:26:42.794514 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:26:42.794515 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 06:26:42.794522 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:26:42.795561 (+ 1039us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:26:42.795885 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:26:42.795886 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:26:42.795886 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:26:42.795934 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:26:42.796151 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:26:42.796151 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:26:42.796151 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:26:42.796248 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:26:42.796294 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0806 06:26:42.796512 (+ 218us) server_negotiation.cc:299] Negotiation successful
0806 06:26:42.796580 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260806 06:26:42.796808 27966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:42.788768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:40387 (local address 127.21.12.131:44165)
0806 06:26:42.788926 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0806 06:26:42.788962 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0806 06:26:42.789493 (+ 531us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:26:42.789721 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:26:42.789725 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:26:42.790021 (+ 296us) client_negotiation.cc:189] Negotiated authn=SASL
0806 06:26:42.790624 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:42.790629 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:42.791292 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:42.791293 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:42.792160 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:42.792162 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:42.792507 (+ 345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:42.792508 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:42.792635 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:42.793097 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 06:26:42.793106 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0806 06:26:42.794435 (+ 1329us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 06:26:42.795632 (+ 1197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:26:42.795638 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:26:42.795640 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:26:42.795834 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:26:42.795997 (+ 163us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:26:42.795998 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:26:42.795998 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:26:42.796103 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:26:42.796291 (+ 188us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 06:26:42.796294 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 06:26:42.796395 (+ 101us) client_negotiation.cc:769] Sending connection context
0806 06:26:42.796497 (+ 102us) client_negotiation.cc:240] Negotiation successful
0806 06:26:42.796627 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":78,"threads_started":1}
I20260806 06:26:42.797242 27963 heartbeater.cc:347] Connected to a master server at 127.21.12.190:40387
I20260806 06:26:42.797327 27963 heartbeater.cc:464] Registering TS with master...
I20260806 06:26:42.797462 27963 heartbeater.cc:511] Master 127.21.12.190:40387 requested a full tablet report, sending...
I20260806 06:26:42.797803 27276 ts_manager.cc:194] Registered new tserver with Master: 36cf9124d6a74980ae20eec5a5995758 (127.21.12.131:37959)
I20260806 06:26:42.798177 27276 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.12.131@KRBTEST.COM'} at 127.21.12.131:44165
I20260806 06:26:42.800894 21554 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 06:26:42.806948 27327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:42.802033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46518 (local address 127.21.12.190:40387)
0806 06:26:42.802071 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0806 06:26:42.802073 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:26:42.802168 (+ 95us) server_negotiation.cc:407] Connection header received
0806 06:26:42.802206 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:26:42.802206 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:26:42.802214 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:26:42.802255 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:26:42.802752 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:42.803188 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:42.803852 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:42.804032 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:42.804081 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:42.804967 (+ 886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:26:42.804968 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:26:42.804968 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 06:26:42.804974 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:26:42.806075 (+ 1101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:26:42.806392 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:26:42.806394 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:26:42.806394 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:26:42.806445 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:26:42.806593 (+ 148us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:26:42.806593 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:26:42.806593 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:26:42.806708 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:26:42.806774 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0806 06:26:42.806839 (+ 65us) server_negotiation.cc:299] Negotiation successful
0806 06:26:42.806893 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260806 06:26:42.808188 27276 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46518:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 06:26:42.809729 27276 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 06:26:42.815508 27974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:42.811799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54500 (local address 127.21.12.131:37959)
0806 06:26:42.812243 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0806 06:26:42.812246 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 06:26:42.812254 (+ 8us) server_negotiation.cc:407] Connection header received
0806 06:26:42.812283 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:26:42.812284 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:26:42.812298 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:26:42.812364 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 06:26:42.812507 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:42.813236 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:42.814618 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:42.815149 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:42.815208 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:42.815262 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0806 06:26:42.815398 (+ 136us) server_negotiation.cc:299] Negotiation successful
0806 06:26:42.815434 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":413,"thread_start_us":77,"threads_started":1}
I20260806 06:26:42.815500 27972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:42.811727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.131:37959 (local address 127.0.0.1:54500)
0806 06:26:42.811835 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0806 06:26:42.811878 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0806 06:26:42.811942 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:26:42.812388 (+ 446us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:26:42.812389 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:26:42.812391 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 06:26:42.812437 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:42.812438 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:42.813323 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:42.813324 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:42.814519 (+ 1195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:42.814522 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:42.815228 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:42.815229 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:42.815308 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:42.815321 (+ 13us) client_negotiation.cc:769] Sending connection context
0806 06:26:42.815378 (+ 57us) client_negotiation.cc:240] Negotiation successful
0806 06:26:42.815430 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":80,"thread_start_us":61,"threads_started":1}
I20260806 06:26:42.815819 27976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:42.811727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.130:42297 (local address 127.0.0.1:38820)
0806 06:26:42.811983 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0806 06:26:42.811993 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0806 06:26:42.812035 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:26:42.812302 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:26:42.812305 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:26:42.812307 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 06:26:42.812367 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:42.812369 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:42.813642 (+ 1273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:42.813643 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:42.814763 (+ 1120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:42.814765 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:42.815608 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:42.815609 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:42.815671 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:42.815675 (+ 4us) client_negotiation.cc:769] Sending connection context
0806 06:26:42.815703 (+ 28us) client_negotiation.cc:240] Negotiation successful
0806 06:26:42.815743 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"mutex_wait_us":43,"thread_start_us":53,"threads_started":1}
I20260806 06:26:42.815827 27973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:42.811915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.129:40521 (local address 127.0.0.1:48254)
0806 06:26:42.812076 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0806 06:26:42.812088 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 06:26:42.812139 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:26:42.812540 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:26:42.812541 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:26:42.812542 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 06:26:42.812585 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:42.812586 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:42.813835 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:42.813836 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:42.814706 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:42.814709 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:42.815641 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:42.815642 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:42.815689 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:42.815692 (+ 3us) client_negotiation.cc:769] Sending connection context
0806 06:26:42.815742 (+ 50us) client_negotiation.cc:240] Negotiation successful
0806 06:26:42.815772 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":37,"threads_started":1}
I20260806 06:26:42.815903 27980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:42.812104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48254 (local address 127.21.12.129:40521)
0806 06:26:42.812250 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0806 06:26:42.812255 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 06:26:42.812265 (+ 10us) server_negotiation.cc:407] Connection header received
0806 06:26:42.812301 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:26:42.812303 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:26:42.812317 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:26:42.812659 (+ 342us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 06:26:42.812739 (+ 80us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:42.813753 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:42.814777 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:42.815468 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:42.815551 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:42.815613 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0806 06:26:42.815777 (+ 164us) server_negotiation.cc:299] Negotiation successful
0806 06:26:42.815824 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":73,"threads_started":1}
I20260806 06:26:42.816445 27850 tablet_service.cc:1467] Processing CreateTablet for tablet 5d2d48fdadfa424baf75c27c37c30e88 (DEFAULT_TABLE table=test-table [id=25f0c14eda72495390fd97a94a8d49c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 06:26:42.816479 27456 tablet_service.cc:1467] Processing CreateTablet for tablet 5d2d48fdadfa424baf75c27c37c30e88 (DEFAULT_TABLE table=test-table [id=25f0c14eda72495390fd97a94a8d49c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 06:26:42.816808 27456 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d2d48fdadfa424baf75c27c37c30e88. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 06:26:42.816808 27850 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d2d48fdadfa424baf75c27c37c30e88. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 06:26:42.819762 27978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:42.811908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38820 (local address 127.21.12.130:42297)
0806 06:26:42.812137 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0806 06:26:42.812143 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 06:26:42.812152 (+ 9us) server_negotiation.cc:407] Connection header received
0806 06:26:42.812190 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:26:42.812192 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:26:42.812208 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:26:42.812285 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 06:26:42.812451 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:42.813542 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:42.814829 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:42.815516 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:42.819538 (+ 4022us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:42.819609 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0806 06:26:42.819636 (+ 27us) server_negotiation.cc:299] Negotiation successful
0806 06:26:42.819681 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":197,"thread_start_us":82,"threads_started":1}
I20260806 06:26:42.820112 27981 tablet_bootstrap.cc:492] T 5d2d48fdadfa424baf75c27c37c30e88 P 14e8034244c14f6aa3081e79abab34dd: Bootstrap starting.
I20260806 06:26:42.820392 27652 tablet_service.cc:1467] Processing CreateTablet for tablet 5d2d48fdadfa424baf75c27c37c30e88 (DEFAULT_TABLE table=test-table [id=25f0c14eda72495390fd97a94a8d49c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 06:26:42.820763 27652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d2d48fdadfa424baf75c27c37c30e88. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 06:26:42.820785 27981 tablet_bootstrap.cc:654] T 5d2d48fdadfa424baf75c27c37c30e88 P 14e8034244c14f6aa3081e79abab34dd: Neither blocks nor log segments found. Creating new log.
I20260806 06:26:42.821352 27981 log.cc:824] T 5d2d48fdadfa424baf75c27c37c30e88 P 14e8034244c14f6aa3081e79abab34dd: Log is configured to *not* fsync() on all Append() calls
I20260806 06:26:42.821709 27982 tablet_bootstrap.cc:492] T 5d2d48fdadfa424baf75c27c37c30e88 P 36cf9124d6a74980ae20eec5a5995758: Bootstrap starting.
I20260806 06:26:42.822074 27981 tablet_bootstrap.cc:492] T 5d2d48fdadfa424baf75c27c37c30e88 P 14e8034244c14f6aa3081e79abab34dd: No bootstrap required, opened a new log
I20260806 06:26:42.822135 27981 ts_tablet_manager.cc:1397] T 5d2d48fdadfa424baf75c27c37c30e88 P 14e8034244c14f6aa3081e79abab34dd: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260806 06:26:42.822352 27982 tablet_bootstrap.cc:654] T 5d2d48fdadfa424baf75c27c37c30e88 P 36cf9124d6a74980ae20eec5a5995758: Neither blocks nor log segments found. Creating new log.
I20260806 06:26:42.822992 27982 log.cc:824] T 5d2d48fdadfa424baf75c27c37c30e88 P 36cf9124d6a74980ae20eec5a5995758: Log is configured to *not* fsync() on all Append() calls
I20260806 06:26:42.823886 27982 tablet_bootstrap.cc:492] T 5d2d48fdadfa424baf75c27c37c30e88 P 36cf9124d6a74980ae20eec5a5995758: No bootstrap required, opened a new log
I20260806 06:26:42.823810 27981 raft_consensus.cc:348] T 5d2d48fdadfa424baf75c27c37c30e88 P 14e8034244c14f6aa3081e79abab34dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd8bfd8a67284cee84b9f9074c16404e" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 42297 } } peers { permanent_uuid: "36cf9124d6a74980ae20eec5a5995758" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 37959 } } peers { permanent_uuid: "14e8034244c14f6aa3081e79abab34dd" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 40521 } }
I20260806 06:26:42.823971 27981 raft_consensus.cc:374] T 5d2d48fdadfa424baf75c27c37c30e88 P 14e8034244c14f6aa3081e79abab34dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 06:26:42.823997 27982 ts_tablet_manager.cc:1397] T 5d2d48fdadfa424baf75c27c37c30e88 P 36cf9124d6a74980ae20eec5a5995758: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260806 06:26:42.824047 27981 raft_consensus.cc:729] T 5d2d48fdadfa424baf75c27c37c30e88 P 14e8034244c14f6aa3081e79abab34dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14e8034244c14f6aa3081e79abab34dd, State: Initialized, Role: FOLLOWER
I20260806 06:26:42.824208 27981 consensus_queue.cc:260] T 5d2d48fdadfa424baf75c27c37c30e88 P 14e8034244c14f6aa3081e79abab34dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd8bfd8a67284cee84b9f9074c16404e" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 42297 } } peers { permanent_uuid: "36cf9124d6a74980ae20eec5a5995758" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 37959 } } peers { permanent_uuid: "14e8034244c14f6aa3081e79abab34dd" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 40521 } }
I20260806 06:26:42.824469 27981 ts_tablet_manager.cc:1428] T 5d2d48fdadfa424baf75c27c37c30e88 P 14e8034244c14f6aa3081e79abab34dd: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260806 06:26:42.824501 27559 heartbeater.cc:503] Master 127.21.12.190:40387 was elected leader, sending a full tablet report...
I20260806 06:26:42.825639 27982 raft_consensus.cc:348] T 5d2d48fdadfa424baf75c27c37c30e88 P 36cf9124d6a74980ae20eec5a5995758 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd8bfd8a67284cee84b9f9074c16404e" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 42297 } } peers { permanent_uuid: "36cf9124d6a74980ae20eec5a5995758" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 37959 } } peers { permanent_uuid: "14e8034244c14f6aa3081e79abab34dd" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 40521 } }
I20260806 06:26:42.825757 27982 raft_consensus.cc:374] T 5d2d48fdadfa424baf75c27c37c30e88 P 36cf9124d6a74980ae20eec5a5995758 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 06:26:42.825785 27982 raft_consensus.cc:729] T 5d2d48fdadfa424baf75c27c37c30e88 P 36cf9124d6a74980ae20eec5a5995758 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36cf9124d6a74980ae20eec5a5995758, State: Initialized, Role: FOLLOWER
I20260806 06:26:42.825888 27982 consensus_queue.cc:260] T 5d2d48fdadfa424baf75c27c37c30e88 P 36cf9124d6a74980ae20eec5a5995758 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd8bfd8a67284cee84b9f9074c16404e" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 42297 } } peers { permanent_uuid: "36cf9124d6a74980ae20eec5a5995758" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 37959 } } peers { permanent_uuid: "14e8034244c14f6aa3081e79abab34dd" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 40521 } }
I20260806 06:26:42.826120 27982 ts_tablet_manager.cc:1428] T 5d2d48fdadfa424baf75c27c37c30e88 P 36cf9124d6a74980ae20eec5a5995758: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260806 06:26:42.826139 27963 heartbeater.cc:503] Master 127.21.12.190:40387 was elected leader, sending a full tablet report...
I20260806 06:26:42.826345 27988 tablet_bootstrap.cc:492] T 5d2d48fdadfa424baf75c27c37c30e88 P dd8bfd8a67284cee84b9f9074c16404e: Bootstrap starting.
I20260806 06:26:42.826886 27988 tablet_bootstrap.cc:654] T 5d2d48fdadfa424baf75c27c37c30e88 P dd8bfd8a67284cee84b9f9074c16404e: Neither blocks nor log segments found. Creating new log.
I20260806 06:26:42.827392 27988 log.cc:824] T 5d2d48fdadfa424baf75c27c37c30e88 P dd8bfd8a67284cee84b9f9074c16404e: Log is configured to *not* fsync() on all Append() calls
I20260806 06:26:42.827970 27988 tablet_bootstrap.cc:492] T 5d2d48fdadfa424baf75c27c37c30e88 P dd8bfd8a67284cee84b9f9074c16404e: No bootstrap required, opened a new log
I20260806 06:26:42.828025 27988 ts_tablet_manager.cc:1397] T 5d2d48fdadfa424baf75c27c37c30e88 P dd8bfd8a67284cee84b9f9074c16404e: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260806 06:26:42.829319 27988 raft_consensus.cc:348] T 5d2d48fdadfa424baf75c27c37c30e88 P dd8bfd8a67284cee84b9f9074c16404e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd8bfd8a67284cee84b9f9074c16404e" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 42297 } } peers { permanent_uuid: "36cf9124d6a74980ae20eec5a5995758" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 37959 } } peers { permanent_uuid: "14e8034244c14f6aa3081e79abab34dd" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 40521 } }
I20260806 06:26:42.829433 27988 raft_consensus.cc:374] T 5d2d48fdadfa424baf75c27c37c30e88 P dd8bfd8a67284cee84b9f9074c16404e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 06:26:42.829463 27988 raft_consensus.cc:729] T 5d2d48fdadfa424baf75c27c37c30e88 P dd8bfd8a67284cee84b9f9074c16404e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd8bfd8a67284cee84b9f9074c16404e, State: Initialized, Role: FOLLOWER
I20260806 06:26:42.829562 27988 consensus_queue.cc:260] T 5d2d48fdadfa424baf75c27c37c30e88 P dd8bfd8a67284cee84b9f9074c16404e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd8bfd8a67284cee84b9f9074c16404e" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 42297 } } peers { permanent_uuid: "36cf9124d6a74980ae20eec5a5995758" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 37959 } } peers { permanent_uuid: "14e8034244c14f6aa3081e79abab34dd" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 40521 } }
I20260806 06:26:42.829782 27988 ts_tablet_manager.cc:1428] T 5d2d48fdadfa424baf75c27c37c30e88 P dd8bfd8a67284cee84b9f9074c16404e: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260806 06:26:42.829808 27745 heartbeater.cc:503] Master 127.21.12.190:40387 was elected leader, sending a full tablet report...
W20260806 06:26:42.877632 27746 tablet.cc:2369] T 5d2d48fdadfa424baf75c27c37c30e88 P dd8bfd8a67284cee84b9f9074c16404e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 06:26:42.929104 27993 raft_consensus.cc:482] T 5d2d48fdadfa424baf75c27c37c30e88 P dd8bfd8a67284cee84b9f9074c16404e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 06:26:42.929234 27993 raft_consensus.cc:504] T 5d2d48fdadfa424baf75c27c37c30e88 P dd8bfd8a67284cee84b9f9074c16404e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd8bfd8a67284cee84b9f9074c16404e" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 42297 } } peers { permanent_uuid: "36cf9124d6a74980ae20eec5a5995758" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 37959 } } peers { permanent_uuid: "14e8034244c14f6aa3081e79abab34dd" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 40521 } }
I20260806 06:26:42.929560 27993 leader_election.cc:288] T 5d2d48fdadfa424baf75c27c37c30e88 P dd8bfd8a67284cee84b9f9074c16404e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 36cf9124d6a74980ae20eec5a5995758 (127.21.12.131:37959), 14e8034244c14f6aa3081e79abab34dd (127.21.12.129:40521)
I20260806 06:26:42.933530 27980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:42.929835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.130:42535 (local address 127.21.12.129:40521)
0806 06:26:42.929881 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0806 06:26:42.929883 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:26:42.929894 (+ 11us) server_negotiation.cc:407] Connection header received
0806 06:26:42.929940 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:26:42.929940 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:26:42.929950 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:26:42.929998 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 06:26:42.930191 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:42.930972 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:42.932589 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:42.933171 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:42.933246 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:42.933255 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0806 06:26:42.933416 (+ 161us) server_negotiation.cc:299] Negotiation successful
0806 06:26:42.933457 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260806 06:26:42.933530 27974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:42.929799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.130:40581 (local address 127.21.12.131:37959)
0806 06:26:42.929845 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0806 06:26:42.929847 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:26:42.930053 (+ 206us) server_negotiation.cc:407] Connection header received
0806 06:26:42.930081 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:26:42.930082 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:26:42.930091 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:26:42.930132 (+ 41us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 06:26:42.930270 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:42.930971 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:42.932599 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:42.933171 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:42.933246 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:42.933255 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0806 06:26:42.933416 (+ 161us) server_negotiation.cc:299] Negotiation successful
0806 06:26:42.933457 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260806 06:26:42.933602 27748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:42.929744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.129:40521 (local address 127.21.12.130:42535)
0806 06:26:42.929821 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0806 06:26:42.929834 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 06:26:42.929882 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:26:42.930030 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:26:42.930031 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:26:42.930033 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 06:26:42.930118 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:42.930119 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:42.931082 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:42.931084 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:42.932465 (+ 1381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:42.932468 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:42.933253 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:42.933255 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:42.933346 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:42.933348 (+ 2us) client_negotiation.cc:769] Sending connection context
0806 06:26:42.933405 (+ 57us) client_negotiation.cc:240] Negotiation successful
0806 06:26:42.933464 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54,"mutex_wait_us":76}
I20260806 06:26:42.933604 27994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:42.929744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.131:37959 (local address 127.21.12.130:40581)
0806 06:26:42.929973 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0806 06:26:42.929980 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0806 06:26:42.930029 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:26:42.930149 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:26:42.930151 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:26:42.930152 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 06:26:42.930213 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:42.930214 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:42.931082 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:42.931084 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:42.932465 (+ 1381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:42.932468 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:42.933254 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:42.933255 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:42.933343 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:42.933347 (+ 4us) client_negotiation.cc:769] Sending connection context
0806 06:26:42.933404 (+ 57us) client_negotiation.cc:240] Negotiation successful
0806 06:26:42.933464 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":217,"mutex_wait_us":137,"thread_start_us":107,"threads_started":1}
I20260806 06:26:42.934046 27884 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d2d48fdadfa424baf75c27c37c30e88" candidate_uuid: "dd8bfd8a67284cee84b9f9074c16404e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36cf9124d6a74980ae20eec5a5995758" is_pre_election: true
I20260806 06:26:42.934043 27494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d2d48fdadfa424baf75c27c37c30e88" candidate_uuid: "dd8bfd8a67284cee84b9f9074c16404e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14e8034244c14f6aa3081e79abab34dd" is_pre_election: true
I20260806 06:26:42.934216 27884 raft_consensus.cc:2454] T 5d2d48fdadfa424baf75c27c37c30e88 P 36cf9124d6a74980ae20eec5a5995758 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd8bfd8a67284cee84b9f9074c16404e in term 0.
I20260806 06:26:42.934216 27494 raft_consensus.cc:2454] T 5d2d48fdadfa424baf75c27c37c30e88 P 14e8034244c14f6aa3081e79abab34dd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd8bfd8a67284cee84b9f9074c16404e in term 0.
I20260806 06:26:42.934479 27586 leader_election.cc:302] T 5d2d48fdadfa424baf75c27c37c30e88 P dd8bfd8a67284cee84b9f9074c16404e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 36cf9124d6a74980ae20eec5a5995758, dd8bfd8a67284cee84b9f9074c16404e; no voters:
I20260806 06:26:42.934628 27993 raft_consensus.cc:2781] T 5d2d48fdadfa424baf75c27c37c30e88 P dd8bfd8a67284cee84b9f9074c16404e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 06:26:42.934700 27993 raft_consensus.cc:482] T 5d2d48fdadfa424baf75c27c37c30e88 P dd8bfd8a67284cee84b9f9074c16404e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 06:26:42.934723 27993 raft_consensus.cc:3037] T 5d2d48fdadfa424baf75c27c37c30e88 P dd8bfd8a67284cee84b9f9074c16404e [term 0 FOLLOWER]: Advancing to term 1
I20260806 06:26:42.935376 27993 raft_consensus.cc:504] T 5d2d48fdadfa424baf75c27c37c30e88 P dd8bfd8a67284cee84b9f9074c16404e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd8bfd8a67284cee84b9f9074c16404e" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 42297 } } peers { permanent_uuid: "36cf9124d6a74980ae20eec5a5995758" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 37959 } } peers { permanent_uuid: "14e8034244c14f6aa3081e79abab34dd" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 40521 } }
I20260806 06:26:42.935504 27993 leader_election.cc:288] T 5d2d48fdadfa424baf75c27c37c30e88 P dd8bfd8a67284cee84b9f9074c16404e [CANDIDATE]: Term 1 election: Requested vote from peers 36cf9124d6a74980ae20eec5a5995758 (127.21.12.131:37959), 14e8034244c14f6aa3081e79abab34dd (127.21.12.129:40521)
I20260806 06:26:42.935678 27884 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d2d48fdadfa424baf75c27c37c30e88" candidate_uuid: "dd8bfd8a67284cee84b9f9074c16404e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36cf9124d6a74980ae20eec5a5995758"
I20260806 06:26:42.935706 27494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d2d48fdadfa424baf75c27c37c30e88" candidate_uuid: "dd8bfd8a67284cee84b9f9074c16404e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14e8034244c14f6aa3081e79abab34dd"
I20260806 06:26:42.935760 27884 raft_consensus.cc:3037] T 5d2d48fdadfa424baf75c27c37c30e88 P 36cf9124d6a74980ae20eec5a5995758 [term 0 FOLLOWER]: Advancing to term 1
I20260806 06:26:42.935777 27494 raft_consensus.cc:3037] T 5d2d48fdadfa424baf75c27c37c30e88 P 14e8034244c14f6aa3081e79abab34dd [term 0 FOLLOWER]: Advancing to term 1
I20260806 06:26:42.936465 27884 raft_consensus.cc:2454] T 5d2d48fdadfa424baf75c27c37c30e88 P 36cf9124d6a74980ae20eec5a5995758 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd8bfd8a67284cee84b9f9074c16404e in term 1.
I20260806 06:26:42.936468 27494 raft_consensus.cc:2454] T 5d2d48fdadfa424baf75c27c37c30e88 P 14e8034244c14f6aa3081e79abab34dd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd8bfd8a67284cee84b9f9074c16404e in term 1.
I20260806 06:26:42.936657 27590 leader_election.cc:302] T 5d2d48fdadfa424baf75c27c37c30e88 P dd8bfd8a67284cee84b9f9074c16404e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14e8034244c14f6aa3081e79abab34dd, dd8bfd8a67284cee84b9f9074c16404e; no voters:
I20260806 06:26:42.936738 27993 raft_consensus.cc:2781] T 5d2d48fdadfa424baf75c27c37c30e88 P dd8bfd8a67284cee84b9f9074c16404e [term 1 FOLLOWER]: Leader election won for term 1
I20260806 06:26:42.936895 27993 raft_consensus.cc:686] T 5d2d48fdadfa424baf75c27c37c30e88 P dd8bfd8a67284cee84b9f9074c16404e [term 1 LEADER]: Becoming Leader. State: Replica: dd8bfd8a67284cee84b9f9074c16404e, State: Running, Role: LEADER
I20260806 06:26:42.936981 27993 consensus_queue.cc:237] T 5d2d48fdadfa424baf75c27c37c30e88 P dd8bfd8a67284cee84b9f9074c16404e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd8bfd8a67284cee84b9f9074c16404e" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 42297 } } peers { permanent_uuid: "36cf9124d6a74980ae20eec5a5995758" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 37959 } } peers { permanent_uuid: "14e8034244c14f6aa3081e79abab34dd" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 40521 } }
I20260806 06:26:42.937798 27276 catalog_manager.cc:5495] T 5d2d48fdadfa424baf75c27c37c30e88 P dd8bfd8a67284cee84b9f9074c16404e reported cstate change: term changed from 0 to 1, leader changed from <none> to dd8bfd8a67284cee84b9f9074c16404e (127.21.12.130). New cstate: current_term: 1 leader_uuid: "dd8bfd8a67284cee84b9f9074c16404e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd8bfd8a67284cee84b9f9074c16404e" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 42297 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "36cf9124d6a74980ae20eec5a5995758" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 37959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "14e8034244c14f6aa3081e79abab34dd" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 40521 } health_report { overall_health: UNKNOWN } } }
I20260806 06:26:42.958828 27978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:42.956661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38834 (local address 127.21.12.130:42297)
0806 06:26:42.956708 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0806 06:26:42.956710 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:26:42.956717 (+ 7us) server_negotiation.cc:407] Connection header received
0806 06:26:42.956749 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:26:42.956750 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:26:42.956767 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:26:42.956829 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 06:26:42.956934 (+ 105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:42.957387 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:42.958110 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:42.958312 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:42.958375 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:42.958567 (+ 192us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 06:26:42.958663 (+ 96us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 06:26:42.958722 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0806 06:26:42.958737 (+ 15us) server_negotiation.cc:299] Negotiation successful
0806 06:26:42.958754 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260806 06:26:42.960500 27884 raft_consensus.cc:1261] T 5d2d48fdadfa424baf75c27c37c30e88 P 36cf9124d6a74980ae20eec5a5995758 [term 1 FOLLOWER]: Refusing update from remote peer dd8bfd8a67284cee84b9f9074c16404e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 06:26:42.960500 27494 raft_consensus.cc:1261] T 5d2d48fdadfa424baf75c27c37c30e88 P 14e8034244c14f6aa3081e79abab34dd [term 1 FOLLOWER]: Refusing update from remote peer dd8bfd8a67284cee84b9f9074c16404e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 06:26:42.960877 27995 consensus_queue.cc:1035] T 5d2d48fdadfa424baf75c27c37c30e88 P dd8bfd8a67284cee84b9f9074c16404e [LEADER]: Connected to new peer: Peer: permanent_uuid: "36cf9124d6a74980ae20eec5a5995758" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 37959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 06:26:42.961004 27993 consensus_queue.cc:1035] T 5d2d48fdadfa424baf75c27c37c30e88 P dd8bfd8a67284cee84b9f9074c16404e [LEADER]: Connected to new peer: Peer: permanent_uuid: "14e8034244c14f6aa3081e79abab34dd" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 40521 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 06:26:42.963858 28000 mvcc.cc:205] Tried to move back new op lower bound from 7315446181723729920 to 7315446181630484480. Current Snapshot: MvccSnapshot[applied={T|T < 7315446181723729920}]
I20260806 06:26:42.964077 27999 mvcc.cc:205] Tried to move back new op lower bound from 7315446181723729920 to 7315446181630484480. Current Snapshot: MvccSnapshot[applied={T|T < 7315446181723729920}]
I20260806 06:26:42.965732 27275 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46518:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 06:26:42.965830 27275 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46518:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 06:26:42.966559 27275 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1b404aadc7de4ea0b15dde51d60d11a1: Sending DeleteTablet for 3 replicas of tablet 5d2d48fdadfa424baf75c27c37c30e88
I20260806 06:26:42.966846 27652 tablet_service.cc:1514] Processing DeleteTablet for tablet 5d2d48fdadfa424baf75c27c37c30e88 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 06:26:42 UTC) from {username='kudu', principal='kudu/127.21.12.190@KRBTEST.COM'} at 127.0.0.1:38820
I20260806 06:26:42.966872 27456 tablet_service.cc:1514] Processing DeleteTablet for tablet 5d2d48fdadfa424baf75c27c37c30e88 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 06:26:42 UTC) from {username='kudu', principal='kudu/127.21.12.190@KRBTEST.COM'} at 127.0.0.1:48254
I20260806 06:26:42.967137 27850 tablet_service.cc:1514] Processing DeleteTablet for tablet 5d2d48fdadfa424baf75c27c37c30e88 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 06:26:42 UTC) from {username='kudu', principal='kudu/127.21.12.190@KRBTEST.COM'} at 127.0.0.1:54500
I20260806 06:26:42.967332 28013 tablet_replica.cc:330] T 5d2d48fdadfa424baf75c27c37c30e88 P 36cf9124d6a74980ae20eec5a5995758: stopping tablet replica
I20260806 06:26:42.967365 21554 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHhLhT/build/release/bin/kudu with pid 27362
I20260806 06:26:42.967455 28013 raft_consensus.cc:2229] T 5d2d48fdadfa424baf75c27c37c30e88 P 36cf9124d6a74980ae20eec5a5995758 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 06:26:42.967556 28013 raft_consensus.cc:2258] T 5d2d48fdadfa424baf75c27c37c30e88 P 36cf9124d6a74980ae20eec5a5995758 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 06:26:42.981173 28011 tablet_replica.cc:330] T 5d2d48fdadfa424baf75c27c37c30e88 P dd8bfd8a67284cee84b9f9074c16404e: stopping tablet replica
I20260806 06:26:42.981321 28011 raft_consensus.cc:2229] T 5d2d48fdadfa424baf75c27c37c30e88 P dd8bfd8a67284cee84b9f9074c16404e [term 1 LEADER]: Raft consensus shutting down.
I20260806 06:26:42.981446 28011 raft_consensus.cc:2258] T 5d2d48fdadfa424baf75c27c37c30e88 P dd8bfd8a67284cee84b9f9074c16404e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 06:26:42.982447 28011 ts_tablet_manager.cc:1905] T 5d2d48fdadfa424baf75c27c37c30e88 P dd8bfd8a67284cee84b9f9074c16404e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 06:26:42.986867 28013 ts_tablet_manager.cc:1905] T 5d2d48fdadfa424baf75c27c37c30e88 P 36cf9124d6a74980ae20eec5a5995758: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 06:26:42.988497 28011 ts_tablet_manager.cc:1918] T 5d2d48fdadfa424baf75c27c37c30e88 P dd8bfd8a67284cee84b9f9074c16404e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 06:26:42.988579 28011 log.cc:1196] T 5d2d48fdadfa424baf75c27c37c30e88 P dd8bfd8a67284cee84b9f9074c16404e: Deleting WAL directory at /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785997581049705-21554-0/minicluster-data/ts-1/wal/wals/5d2d48fdadfa424baf75c27c37c30e88
I20260806 06:26:42.988840 28011 ts_tablet_manager.cc:1939] T 5d2d48fdadfa424baf75c27c37c30e88 P dd8bfd8a67284cee84b9f9074c16404e: Deleting consensus metadata
I20260806 06:26:42.989312 27258 catalog_manager.cc:4841] TS dd8bfd8a67284cee84b9f9074c16404e (127.21.12.130:42297): tablet 5d2d48fdadfa424baf75c27c37c30e88 (table test-table [id=25f0c14eda72495390fd97a94a8d49c8]) successfully deleted
I20260806 06:26:42.990708 28013 ts_tablet_manager.cc:1918] T 5d2d48fdadfa424baf75c27c37c30e88 P 36cf9124d6a74980ae20eec5a5995758: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 06:26:42.990785 28013 log.cc:1196] T 5d2d48fdadfa424baf75c27c37c30e88 P 36cf9124d6a74980ae20eec5a5995758: Deleting WAL directory at /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785997581049705-21554-0/minicluster-data/ts-2/wal/wals/5d2d48fdadfa424baf75c27c37c30e88
I20260806 06:26:42.991014 28013 ts_tablet_manager.cc:1939] T 5d2d48fdadfa424baf75c27c37c30e88 P 36cf9124d6a74980ae20eec5a5995758: Deleting consensus metadata
I20260806 06:26:42.991449 27256 catalog_manager.cc:4841] TS 36cf9124d6a74980ae20eec5a5995758 (127.21.12.131:37959): tablet 5d2d48fdadfa424baf75c27c37c30e88 (table test-table [id=25f0c14eda72495390fd97a94a8d49c8]) successfully deleted
W20260806 06:26:42.992683 27258 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.12.129:40521 (error 108)
I20260806 06:26:42.993012 21554 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHhLhT/build/release/bin/kudu with pid 27567
I20260806 06:26:42.993161 27973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:42.993033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.129:40521 (local address 127.0.0.1:48264)
0806 06:26:42.993087 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0806 06:26:42.993128 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.129:40521: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260806 06:26:42.994483 27258 catalog_manager.cc:4584] TS 14e8034244c14f6aa3081e79abab34dd (127.21.12.129:40521): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5d2d48fdadfa424baf75c27c37c30e88: Network error: Client connection negotiation failed: client connection to 127.21.12.129:40521: connect: Connection refused (error 111)
I20260806 06:26:42.998795 21554 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHhLhT/build/release/bin/kudu with pid 27756
I20260806 06:26:43.012846 21554 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHhLhT/build/release/bin/kudu with pid 27229
2026-08-06T06:26:43Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2078 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-08-06T06:26:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T06:26:43Z Disabled control of system clock
I20260806 06:26:43.049146 21554 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHhLhT/build/release/bin/kudu
/tmp/dist-test-taskFHhLhT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785997581049705-21554-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785997581049705-21554-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785997581049705-21554-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785997581049705-21554-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.12.190:37381
--webserver_interface=127.21.12.190
--webserver_port=0
--builtin_ntp_servers=127.21.12.148:32931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.12.190:37381
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260806 06:26:43.127162 28019 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 06:26:43.127358 28019 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 06:26:43.127386 28019 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 06:26:43.128502 28019 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 06:26:43.128542 28019 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 06:26:43.128559 28019 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 06:26:43.128575 28019 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 06:26:43.128592 28019 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 06:26:43.129925 28019 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.148:32931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785997581049705-21554-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785997581049705-21554-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.12.190:37381
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.12.190:37381
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785997581049705-21554-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.12.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28019
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785997581049705-21554-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 05:11:02 UTC on ccycloud.quasar-hvzbbz.root.comops.site
build id 4569617
I20260806 06:26:43.130159 28019 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 06:26:43.130465 28019 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 06:26:43.135125 28024 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:26:43.135114 28025 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:26:43.135115 28029 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:26:43.135380 28019 server_base.cc:1034] running on GCE node
I20260806 06:26:43.135615 28019 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 06:26:43.135941 28019 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 06:26:43.137107 28019 hybrid_clock.cc:648] HybridClock initialized: now 1785997603137087 us; error 31 us; skew 500 ppm
I20260806 06:26:43.138357 28019 webserver.cc:460] Webserver started at http://127.21.12.190:40423/ using document root <none> and password file <none>
I20260806 06:26:43.138608 28019 fs_manager.cc:359] Metadata directory not provided
I20260806 06:26:43.138662 28019 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 06:26:43.138800 28019 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 06:26:43.139860 28019 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785997581049705-21554-0/minicluster-data/master-0/data/instance:
uuid: "4cef0803e77b40bf8407c0022af7698a"
format_stamp: "Formatted at 2026-08-06 06:26:43 on dist-test-slave-4tdj"
server_key: "e5b792c4fc82a90672bea810a87d275a"
server_key_iv: "d6d5f38677e7607f9b45dd1f9566c58e"
server_key_version: "encryptionkey@0"
I20260806 06:26:43.140221 28019 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785997581049705-21554-0/minicluster-data/master-0/wal/instance:
uuid: "4cef0803e77b40bf8407c0022af7698a"
format_stamp: "Formatted at 2026-08-06 06:26:43 on dist-test-slave-4tdj"
server_key: "e5b792c4fc82a90672bea810a87d275a"
server_key_iv: "d6d5f38677e7607f9b45dd1f9566c58e"
server_key_version: "encryptionkey@0"
I20260806 06:26:43.141448 28019 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260806 06:26:43.142134 28041 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 06:26:43.142285 28019 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260806 06:26:43.142410 28019 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785997581049705-21554-0/minicluster-data/master-0/data,/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785997581049705-21554-0/minicluster-data/master-0/wal
uuid: "4cef0803e77b40bf8407c0022af7698a"
format_stamp: "Formatted at 2026-08-06 06:26:43 on dist-test-slave-4tdj"
server_key: "e5b792c4fc82a90672bea810a87d275a"
server_key_iv: "d6d5f38677e7607f9b45dd1f9566c58e"
server_key_version: "encryptionkey@0"
I20260806 06:26:43.142480 28019 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785997581049705-21554-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785997581049705-21554-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785997581049705-21554-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 06:26:43.145819 28019 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 06:26:43.146073 28019 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 06:26:43.146204 28019 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 06:26:43.150049 28019 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.190:37381
I20260806 06:26:43.150079 28115 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.190:37381 every 8 connection(s)
I20260806 06:26:43.150462 28019 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785997581049705-21554-0/minicluster-data/master-0/data/info.pb
I20260806 06:26:43.151207 28117 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 06:26:43.152859 21554 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHhLhT/build/release/bin/kudu as pid 28019
I20260806 06:26:43.152957 21554 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785997581049705-21554-0/minicluster-data/master-0/wal/instance
I20260806 06:26:43.153122 21554 external_mini_cluster.cc:1428] Setting key cf9db8eed6a8832c5894823a82570d70
I20260806 06:26:43.154244 28117 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4cef0803e77b40bf8407c0022af7698a: Bootstrap starting.
I20260806 06:26:43.154835 28117 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4cef0803e77b40bf8407c0022af7698a: Neither blocks nor log segments found. Creating new log.
I20260806 06:26:43.155117 28117 log.cc:824] T 00000000000000000000000000000000 P 4cef0803e77b40bf8407c0022af7698a: Log is configured to *not* fsync() on all Append() calls
I20260806 06:26:43.155722 28124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:43.153624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54870 (local address 127.21.12.190:37381)
0806 06:26:43.153801 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0806 06:26:43.153808 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 06:26:43.153831 (+ 23us) server_negotiation.cc:407] Connection header received
0806 06:26:43.154219 (+ 388us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:26:43.154226 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:26:43.154262 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:26:43.154415 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:26:43.155275 (+ 860us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:26:43.155279 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:26:43.155279 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 06:26:43.155284 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:26:43.155335 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 06:26:43.155346 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:26:43.155401 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0806 06:26:43.155481 (+ 80us) server_negotiation.cc:299] Negotiation successful
0806 06:26:43.155538 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":62,"threads_started":1}
I20260806 06:26:43.155922 28117 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4cef0803e77b40bf8407c0022af7698a: No bootstrap required, opened a new log
I20260806 06:26:43.157368 28117 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4cef0803e77b40bf8407c0022af7698a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cef0803e77b40bf8407c0022af7698a" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 37381 } }
I20260806 06:26:43.157495 28117 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4cef0803e77b40bf8407c0022af7698a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 06:26:43.157517 28117 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4cef0803e77b40bf8407c0022af7698a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cef0803e77b40bf8407c0022af7698a, State: Initialized, Role: FOLLOWER
I20260806 06:26:43.157632 28117 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4cef0803e77b40bf8407c0022af7698a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cef0803e77b40bf8407c0022af7698a" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 37381 } }
I20260806 06:26:43.157716 28117 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4cef0803e77b40bf8407c0022af7698a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 06:26:43.157748 28117 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4cef0803e77b40bf8407c0022af7698a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 06:26:43.157793 28117 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4cef0803e77b40bf8407c0022af7698a [term 0 FOLLOWER]: Advancing to term 1
I20260806 06:26:43.158586 28117 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4cef0803e77b40bf8407c0022af7698a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cef0803e77b40bf8407c0022af7698a" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 37381 } }
I20260806 06:26:43.158680 28117 leader_election.cc:302] T 00000000000000000000000000000000 P 4cef0803e77b40bf8407c0022af7698a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4cef0803e77b40bf8407c0022af7698a; no voters:
I20260806 06:26:43.158844 28117 leader_election.cc:288] T 00000000000000000000000000000000 P 4cef0803e77b40bf8407c0022af7698a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 06:26:43.158890 28126 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4cef0803e77b40bf8407c0022af7698a [term 1 FOLLOWER]: Leader election won for term 1
I20260806 06:26:43.159076 28126 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4cef0803e77b40bf8407c0022af7698a [term 1 LEADER]: Becoming Leader. State: Replica: 4cef0803e77b40bf8407c0022af7698a, State: Running, Role: LEADER
I20260806 06:26:43.159122 28117 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4cef0803e77b40bf8407c0022af7698a [sys.catalog]: configured and running, proceeding with master startup.
I20260806 06:26:43.159164 28126 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4cef0803e77b40bf8407c0022af7698a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cef0803e77b40bf8407c0022af7698a" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 37381 } }
I20260806 06:26:43.159468 28126 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4cef0803e77b40bf8407c0022af7698a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4cef0803e77b40bf8407c0022af7698a. Latest consensus state: current_term: 1 leader_uuid: "4cef0803e77b40bf8407c0022af7698a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cef0803e77b40bf8407c0022af7698a" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 37381 } } }
I20260806 06:26:43.159500 28127 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4cef0803e77b40bf8407c0022af7698a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4cef0803e77b40bf8407c0022af7698a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cef0803e77b40bf8407c0022af7698a" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 37381 } } }
I20260806 06:26:43.159574 28126 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4cef0803e77b40bf8407c0022af7698a [sys.catalog]: This master's current role is: LEADER
I20260806 06:26:43.159653 28127 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4cef0803e77b40bf8407c0022af7698a [sys.catalog]: This master's current role is: LEADER
I20260806 06:26:43.159874 28139 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 06:26:43.160387 28139 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 06:26:43.161841 28139 catalog_manager.cc:1347] Generated new cluster ID: ca339ea2ff7b4f6891f583169bd9c33a
I20260806 06:26:43.161892 28139 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 06:26:43.170469 28139 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 06:26:43.170615 28139 catalog_manager.cc:1504] Loading token signing keys...
I20260806 06:26:43.176378 28139 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4cef0803e77b40bf8407c0022af7698a: Generated new TSK 0
I20260806 06:26:43.176545 28139 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 06:26:43.180353 21554 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHhLhT/build/release/bin/kudu
/tmp/dist-test-taskFHhLhT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785997581049705-21554-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785997581049705-21554-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785997581049705-21554-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785997581049705-21554-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.12.129:0
--local_ip_for_outbound_sockets=127.21.12.129
--webserver_interface=127.21.12.129
--webserver_port=0
--tserver_master_addrs=127.21.12.190:37381
--builtin_ntp_servers=127.21.12.148:32931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260806 06:26:43.260042 28152 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 06:26:43.260244 28152 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 06:26:43.260273 28152 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 06:26:43.261446 28152 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 06:26:43.261487 28152 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 06:26:43.261520 28152 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.12.129
I20260806 06:26:43.262852 28152 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.148:32931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785997581049705-21554-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785997581049705-21554-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.12.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785997581049705-21554-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.12.129
--webserver_port=0
--tserver_master_addrs=127.21.12.190:37381
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28152
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.12.129
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785997581049705-21554-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 05:11:02 UTC on ccycloud.quasar-hvzbbz.root.comops.site
build id 4569617
I20260806 06:26:43.263077 28152 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 06:26:43.263334 28152 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 06:26:43.268047 28163 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:26:43.268085 28160 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:26:43.268090 28161 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:26:43.268458 28152 server_base.cc:1034] running on GCE node
I20260806 06:26:43.268669 28152 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 06:26:43.268877 28152 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 06:26:43.270020 28152 hybrid_clock.cc:648] HybridClock initialized: now 1785997603270006 us; error 33 us; skew 500 ppm
I20260806 06:26:43.271230 28152 webserver.cc:460] Webserver started at http://127.21.12.129:33311/ using document root <none> and password file <none>
I20260806 06:26:43.271458 28152 fs_manager.cc:359] Metadata directory not provided
I20260806 06:26:43.271504 28152 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 06:26:43.271665 28152 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 06:26:43.272676 28152 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785997581049705-21554-0/minicluster-data/ts-0/data/instance:
uuid: "2d5a27fe74d140d593b27586396f6ce6"
format_stamp: "Formatted at 2026-08-06 06:26:43 on dist-test-slave-4tdj"
server_key: "99fba4fb14798968f0cb6de66d3d6c91"
server_key_iv: "d3e60cde866e3d6cbd6916e6ec710372"
server_key_version: "encryptionkey@0"
I20260806 06:26:43.272993 28152 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785997581049705-21554-0/minicluster-data/ts-0/wal/instance:
uuid: "2d5a27fe74d140d593b27586396f6ce6"
format_stamp: "Formatted at 2026-08-06 06:26:43 on dist-test-slave-4tdj"
server_key: "99fba4fb14798968f0cb6de66d3d6c91"
server_key_iv: "d3e60cde866e3d6cbd6916e6ec710372"
server_key_version: "encryptionkey@0"
I20260806 06:26:43.274219 28152 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260806 06:26:43.274956 28169 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 06:26:43.275101 28152 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260806 06:26:43.275177 28152 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785997581049705-21554-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785997581049705-21554-0/minicluster-data/ts-0/wal
uuid: "2d5a27fe74d140d593b27586396f6ce6"
format_stamp: "Formatted at 2026-08-06 06:26:43 on dist-test-slave-4tdj"
server_key: "99fba4fb14798968f0cb6de66d3d6c91"
server_key_iv: "d3e60cde866e3d6cbd6916e6ec710372"
server_key_version: "encryptionkey@0"
I20260806 06:26:43.275243 28152 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785997581049705-21554-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785997581049705-21554-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785997581049705-21554-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 06:26:43.278343 28152 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 06:26:43.278594 28152 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 06:26:43.278703 28152 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 06:26:43.278900 28152 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 06:26:43.279170 28152 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 06:26:43.279201 28152 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 06:26:43.279220 28152 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 06:26:43.279243 28152 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 06:26:43.284489 28152 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.129:46643
I20260806 06:26:43.284530 28331 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.129:46643 every 8 connection(s)
I20260806 06:26:43.284821 28152 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785997581049705-21554-0/minicluster-data/ts-0/data/info.pb
I20260806 06:26:43.285519 21554 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHhLhT/build/release/bin/kudu as pid 28152
I20260806 06:26:43.285622 21554 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785997581049705-21554-0/minicluster-data/ts-0/wal/instance
I20260806 06:26:43.285773 21554 external_mini_cluster.cc:1428] Setting key b3d18ed13e53a342dae147cc471746bb
I20260806 06:26:43.286680 21554 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHhLhT/build/release/bin/kudu
/tmp/dist-test-taskFHhLhT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785997581049705-21554-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785997581049705-21554-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785997581049705-21554-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785997581049705-21554-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.12.130:0
--local_ip_for_outbound_sockets=127.21.12.130
--webserver_interface=127.21.12.130
--webserver_port=0
--tserver_master_addrs=127.21.12.190:37381
--builtin_ntp_servers=127.21.12.148:32931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260806 06:26:43.287688 28124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:43.285489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.129:49031 (local address 127.21.12.190:37381)
0806 06:26:43.285569 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0806 06:26:43.285571 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:26:43.285742 (+ 171us) server_negotiation.cc:407] Connection header received
0806 06:26:43.286203 (+ 461us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:26:43.286204 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:26:43.286210 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:26:43.286284 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:26:43.287448 (+ 1164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:26:43.287450 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:26:43.287450 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 06:26:43.287455 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:26:43.287473 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 06:26:43.287477 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:26:43.287551 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0806 06:26:43.287610 (+ 59us) server_negotiation.cc:299] Negotiation successful
0806 06:26:43.287623 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260806 06:26:43.287811 28337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:43.285449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:37381 (local address 127.21.12.129:49031)
0806 06:26:43.285621 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0806 06:26:43.285658 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0806 06:26:43.286124 (+ 466us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:26:43.286372 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:26:43.286377 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:26:43.286387 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0806 06:26:43.287298 (+ 911us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 06:26:43.287299 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0806 06:26:43.287325 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 06:26:43.287352 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 06:26:43.287554 (+ 202us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 06:26:43.287556 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 06:26:43.287559 (+ 3us) client_negotiation.cc:769] Sending connection context
0806 06:26:43.287601 (+ 42us) client_negotiation.cc:240] Negotiation successful
0806 06:26:43.287646 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":73,"threads_started":1}
I20260806 06:26:43.288326 28334 heartbeater.cc:347] Connected to a master server at 127.21.12.190:37381
I20260806 06:26:43.288426 28334 heartbeater.cc:464] Registering TS with master...
I20260806 06:26:43.288568 28334 heartbeater.cc:511] Master 127.21.12.190:37381 requested a full tablet report, sending...
I20260806 06:26:43.289021 28061 ts_manager.cc:194] Registered new tserver with Master: 2d5a27fe74d140d593b27586396f6ce6 (127.21.12.129:46643)
W20260806 06:26:43.367123 28338 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 06:26:43.367312 28338 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 06:26:43.367333 28338 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 06:26:43.368404 28338 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 06:26:43.368435 28338 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 06:26:43.368459 28338 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.12.130
I20260806 06:26:43.369712 28338 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.148:32931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785997581049705-21554-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785997581049705-21554-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.12.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785997581049705-21554-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.12.130
--webserver_port=0
--tserver_master_addrs=127.21.12.190:37381
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28338
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.12.130
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785997581049705-21554-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 05:11:02 UTC on ccycloud.quasar-hvzbbz.root.comops.site
build id 4569617
I20260806 06:26:43.369902 28338 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 06:26:43.370172 28338 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 06:26:43.374835 28345 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:26:43.374900 28347 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:26:43.374831 28349 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:26:43.374909 28338 server_base.cc:1034] running on GCE node
I20260806 06:26:43.375231 28338 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 06:26:43.375419 28338 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 06:26:43.376549 28338 hybrid_clock.cc:648] HybridClock initialized: now 1785997603376537 us; error 28 us; skew 500 ppm
I20260806 06:26:43.377722 28338 webserver.cc:460] Webserver started at http://127.21.12.130:33953/ using document root <none> and password file <none>
I20260806 06:26:43.377933 28338 fs_manager.cc:359] Metadata directory not provided
I20260806 06:26:43.377977 28338 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 06:26:43.378098 28338 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 06:26:43.379165 28338 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785997581049705-21554-0/minicluster-data/ts-1/data/instance:
uuid: "57d6f8f29b3f4792af16b482a70b859f"
format_stamp: "Formatted at 2026-08-06 06:26:43 on dist-test-slave-4tdj"
server_key: "1f8b45ee542f59b7ffcedfdcda2c130e"
server_key_iv: "0a8e3d8137fcd3ce9048696eccf7fbf7"
server_key_version: "encryptionkey@0"
I20260806 06:26:43.379487 28338 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785997581049705-21554-0/minicluster-data/ts-1/wal/instance:
uuid: "57d6f8f29b3f4792af16b482a70b859f"
format_stamp: "Formatted at 2026-08-06 06:26:43 on dist-test-slave-4tdj"
server_key: "1f8b45ee542f59b7ffcedfdcda2c130e"
server_key_iv: "0a8e3d8137fcd3ce9048696eccf7fbf7"
server_key_version: "encryptionkey@0"
I20260806 06:26:43.380790 28338 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260806 06:26:43.381484 28357 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 06:26:43.381621 28338 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260806 06:26:43.381716 28338 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785997581049705-21554-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785997581049705-21554-0/minicluster-data/ts-1/wal
uuid: "57d6f8f29b3f4792af16b482a70b859f"
format_stamp: "Formatted at 2026-08-06 06:26:43 on dist-test-slave-4tdj"
server_key: "1f8b45ee542f59b7ffcedfdcda2c130e"
server_key_iv: "0a8e3d8137fcd3ce9048696eccf7fbf7"
server_key_version: "encryptionkey@0"
I20260806 06:26:43.381774 28338 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785997581049705-21554-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785997581049705-21554-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785997581049705-21554-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 06:26:43.384965 28338 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 06:26:43.385196 28338 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 06:26:43.385320 28338 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 06:26:43.385497 28338 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 06:26:43.385775 28338 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 06:26:43.385808 28338 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 06:26:43.385833 28338 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 06:26:43.385859 28338 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 06:26:43.391265 28338 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.130:41347
I20260806 06:26:43.391299 28521 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.130:41347 every 8 connection(s)
I20260806 06:26:43.391568 28338 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785997581049705-21554-0/minicluster-data/ts-1/data/info.pb
I20260806 06:26:43.391901 21554 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHhLhT/build/release/bin/kudu as pid 28338
I20260806 06:26:43.391992 21554 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785997581049705-21554-0/minicluster-data/ts-1/wal/instance
I20260806 06:26:43.392174 21554 external_mini_cluster.cc:1428] Setting key 35a16fc47e05739dd5e4f5f6f0063924
I20260806 06:26:43.393158 21554 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHhLhT/build/release/bin/kudu
/tmp/dist-test-taskFHhLhT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785997581049705-21554-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785997581049705-21554-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785997581049705-21554-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785997581049705-21554-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.12.131:0
--local_ip_for_outbound_sockets=127.21.12.131
--webserver_interface=127.21.12.131
--webserver_port=0
--tserver_master_addrs=127.21.12.190:37381
--builtin_ntp_servers=127.21.12.148:32931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260806 06:26:43.394610 28124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:43.392194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.130:55429 (local address 127.21.12.190:37381)
0806 06:26:43.392239 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0806 06:26:43.392241 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:26:43.392519 (+ 278us) server_negotiation.cc:407] Connection header received
0806 06:26:43.393127 (+ 608us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:26:43.393128 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:26:43.393136 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:26:43.393206 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:26:43.394346 (+ 1140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:26:43.394348 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:26:43.394348 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 06:26:43.394353 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:26:43.394370 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 06:26:43.394375 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:26:43.394448 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0806 06:26:43.394522 (+ 74us) server_negotiation.cc:299] Negotiation successful
0806 06:26:43.394537 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260806 06:26:43.394793 28524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:43.392207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:37381 (local address 127.21.12.130:55429)
0806 06:26:43.392400 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0806 06:26:43.392440 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0806 06:26:43.393019 (+ 579us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:26:43.393271 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:26:43.393276 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:26:43.393287 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0806 06:26:43.394186 (+ 899us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 06:26:43.394186 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0806 06:26:43.394213 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 06:26:43.394239 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 06:26:43.394450 (+ 211us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 06:26:43.394455 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 06:26:43.394458 (+ 3us) client_negotiation.cc:769] Sending connection context
0806 06:26:43.394535 (+ 77us) client_negotiation.cc:240] Negotiation successful
0806 06:26:43.394600 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":89,"threads_started":1}
I20260806 06:26:43.395277 28522 heartbeater.cc:347] Connected to a master server at 127.21.12.190:37381
I20260806 06:26:43.395363 28522 heartbeater.cc:464] Registering TS with master...
I20260806 06:26:43.395500 28522 heartbeater.cc:511] Master 127.21.12.190:37381 requested a full tablet report, sending...
I20260806 06:26:43.395844 28061 ts_manager.cc:194] Registered new tserver with Master: 57d6f8f29b3f4792af16b482a70b859f (127.21.12.130:41347)
W20260806 06:26:43.474197 28525 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 06:26:43.474439 28525 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 06:26:43.474474 28525 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 06:26:43.475605 28525 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 06:26:43.475642 28525 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 06:26:43.475675 28525 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.12.131
I20260806 06:26:43.476974 28525 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.148:32931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785997581049705-21554-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785997581049705-21554-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.12.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785997581049705-21554-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.12.131
--webserver_port=0
--tserver_master_addrs=127.21.12.190:37381
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28525
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.12.131
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785997581049705-21554-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 05:11:02 UTC on ccycloud.quasar-hvzbbz.root.comops.site
build id 4569617
I20260806 06:26:43.477216 28525 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 06:26:43.477476 28525 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260806 06:26:43.482244 28525 server_base.cc:1034] running on GCE node
W20260806 06:26:43.482223 28534 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:26:43.482188 28532 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:26:43.482184 28539 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:26:43.482623 28525 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 06:26:43.482829 28525 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 06:26:43.483983 28525 hybrid_clock.cc:648] HybridClock initialized: now 1785997603483966 us; error 32 us; skew 500 ppm
I20260806 06:26:43.485188 28525 webserver.cc:460] Webserver started at http://127.21.12.131:42663/ using document root <none> and password file <none>
I20260806 06:26:43.485437 28525 fs_manager.cc:359] Metadata directory not provided
I20260806 06:26:43.485481 28525 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 06:26:43.485581 28525 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 06:26:43.486603 28525 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785997581049705-21554-0/minicluster-data/ts-2/data/instance:
uuid: "5e22e0ce55394248a1d9f866db8258f7"
format_stamp: "Formatted at 2026-08-06 06:26:43 on dist-test-slave-4tdj"
server_key: "6757c4fbecf4b4a78c4e0b82405a6e12"
server_key_iv: "4ac0966a9d9c093f43b5d29259c12792"
server_key_version: "encryptionkey@0"
I20260806 06:26:43.486888 28525 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785997581049705-21554-0/minicluster-data/ts-2/wal/instance:
uuid: "5e22e0ce55394248a1d9f866db8258f7"
format_stamp: "Formatted at 2026-08-06 06:26:43 on dist-test-slave-4tdj"
server_key: "6757c4fbecf4b4a78c4e0b82405a6e12"
server_key_iv: "4ac0966a9d9c093f43b5d29259c12792"
server_key_version: "encryptionkey@0"
I20260806 06:26:43.488020 28525 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260806 06:26:43.488662 28550 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 06:26:43.488842 28525 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260806 06:26:43.488929 28525 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785997581049705-21554-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785997581049705-21554-0/minicluster-data/ts-2/wal
uuid: "5e22e0ce55394248a1d9f866db8258f7"
format_stamp: "Formatted at 2026-08-06 06:26:43 on dist-test-slave-4tdj"
server_key: "6757c4fbecf4b4a78c4e0b82405a6e12"
server_key_iv: "4ac0966a9d9c093f43b5d29259c12792"
server_key_version: "encryptionkey@0"
I20260806 06:26:43.488996 28525 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785997581049705-21554-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785997581049705-21554-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785997581049705-21554-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 06:26:43.492213 28525 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 06:26:43.492451 28525 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 06:26:43.492553 28525 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 06:26:43.492735 28525 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 06:26:43.493007 28525 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 06:26:43.493041 28525 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 06:26:43.493070 28525 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 06:26:43.493093 28525 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 06:26:43.498533 28525 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.131:40139
I20260806 06:26:43.498562 28708 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.131:40139 every 8 connection(s)
I20260806 06:26:43.498850 28525 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785997581049705-21554-0/minicluster-data/ts-2/data/info.pb
I20260806 06:26:43.501405 28124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:43.499515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.131:46745 (local address 127.21.12.190:37381)
0806 06:26:43.499594 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0806 06:26:43.499596 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:26:43.499723 (+ 127us) server_negotiation.cc:407] Connection header received
0806 06:26:43.500154 (+ 431us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:26:43.500156 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:26:43.500161 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:26:43.500200 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:26:43.501246 (+ 1046us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:26:43.501247 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:26:43.501247 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 06:26:43.501249 (+ 2us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:26:43.501259 (+ 10us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 06:26:43.501262 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:26:43.501295 (+ 33us) server_negotiation.cc:1035] Waiting for connection context
0806 06:26:43.501342 (+ 47us) server_negotiation.cc:299] Negotiation successful
0806 06:26:43.501352 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260806 06:26:43.501564 28713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:43.499451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:37381 (local address 127.21.12.131:46745)
0806 06:26:43.499617 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0806 06:26:43.499648 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0806 06:26:43.500065 (+ 417us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:26:43.500250 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:26:43.500253 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:26:43.500261 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0806 06:26:43.501130 (+ 869us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 06:26:43.501131 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0806 06:26:43.501157 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 06:26:43.501186 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 06:26:43.501300 (+ 114us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 06:26:43.501302 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 06:26:43.501304 (+ 2us) client_negotiation.cc:769] Sending connection context
0806 06:26:43.501348 (+ 44us) client_negotiation.cc:240] Negotiation successful
0806 06:26:43.501391 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":69,"threads_started":1}
I20260806 06:26:43.501983 28709 heartbeater.cc:347] Connected to a master server at 127.21.12.190:37381
I20260806 06:26:43.502061 28709 heartbeater.cc:464] Registering TS with master...
I20260806 06:26:43.502162 28709 heartbeater.cc:511] Master 127.21.12.190:37381 requested a full tablet report, sending...
I20260806 06:26:43.502508 28061 ts_manager.cc:194] Registered new tserver with Master: 5e22e0ce55394248a1d9f866db8258f7 (127.21.12.131:40139)
I20260806 06:26:43.508601 21554 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHhLhT/build/release/bin/kudu as pid 28525
I20260806 06:26:43.508687 21554 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785997581049705-21554-0/minicluster-data/ts-2/wal/instance
I20260806 06:26:43.508857 21554 external_mini_cluster.cc:1428] Setting key 4d7deed1c6de9e8da66421a86a704438
I20260806 06:26:43.509760 21554 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 06:26:43.511958 28124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:43.511093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54884 (local address 127.21.12.190:37381)
0806 06:26:43.511145 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0806 06:26:43.511147 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:26:43.511225 (+ 78us) server_negotiation.cc:407] Connection header received
0806 06:26:43.511280 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:26:43.511281 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:26:43.511285 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:26:43.511321 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:26:43.511786 (+ 465us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:26:43.511787 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:26:43.511787 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 06:26:43.511790 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:26:43.511802 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 06:26:43.511805 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:26:43.511845 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0806 06:26:43.511881 (+ 36us) server_negotiation.cc:299] Negotiation successful
0806 06:26:43.511891 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260806 06:26:43.513230 28061 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54884:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 06:26:43.514742 28061 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 06:26:43.518111 28719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:43.516834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.129:46643 (local address 127.0.0.1:33684)
0806 06:26:43.516967 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0806 06:26:43.516975 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0806 06:26:43.517005 (+ 30us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:26:43.517365 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:26:43.517366 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:26:43.517369 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0806 06:26:43.517733 (+ 364us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 06:26:43.517734 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0806 06:26:43.517750 (+ 16us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 06:26:43.517773 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 06:26:43.517979 (+ 206us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 06:26:43.517985 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 06:26:43.517988 (+ 3us) client_negotiation.cc:769] Sending connection context
0806 06:26:43.518033 (+ 45us) client_negotiation.cc:240] Negotiation successful
0806 06:26:43.518045 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":68,"threads_started":1}
I20260806 06:26:43.518230 28724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:43.516642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.130:41347 (local address 127.0.0.1:60996)
0806 06:26:43.516960 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0806 06:26:43.516970 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0806 06:26:43.517033 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:26:43.517534 (+ 501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:26:43.517535 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:26:43.517537 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0806 06:26:43.517855 (+ 318us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 06:26:43.517855 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0806 06:26:43.517862 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 06:26:43.517873 (+ 11us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 06:26:43.518118 (+ 245us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 06:26:43.518119 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 06:26:43.518119 (+ 0us) client_negotiation.cc:769] Sending connection context
0806 06:26:43.518167 (+ 48us) client_negotiation.cc:240] Negotiation successful
0806 06:26:43.518179 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":78,"threads_started":1}
I20260806 06:26:43.518308 28723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:43.516753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60996 (local address 127.21.12.130:41347)
0806 06:26:43.516931 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0806 06:26:43.516935 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 06:26:43.517087 (+ 152us) server_negotiation.cc:407] Connection header received
0806 06:26:43.517114 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:26:43.517116 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:26:43.517125 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:26:43.517184 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:26:43.517961 (+ 777us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:26:43.517969 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:26:43.517969 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 06:26:43.517973 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:26:43.518005 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 06:26:43.518044 (+ 39us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:26:43.518094 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0806 06:26:43.518185 (+ 91us) server_negotiation.cc:299] Negotiation successful
0806 06:26:43.518203 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":73,"threads_started":1}
I20260806 06:26:43.518395 28720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:43.516630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.131:40139 (local address 127.0.0.1:58858)
0806 06:26:43.516799 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0806 06:26:43.516839 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0806 06:26:43.516892 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:26:43.517233 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:26:43.517240 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:26:43.517246 (+ 6us) client_negotiation.cc:189] Negotiated authn=SASL
0806 06:26:43.518027 (+ 781us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 06:26:43.518027 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0806 06:26:43.518033 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 06:26:43.518045 (+ 12us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 06:26:43.518264 (+ 219us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 06:26:43.518264 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 06:26:43.518265 (+ 1us) client_negotiation.cc:769] Sending connection context
0806 06:26:43.518322 (+ 57us) client_negotiation.cc:240] Negotiation successful
0806 06:26:43.518334 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":67,"threads_started":1}
I20260806 06:26:43.518569 28721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:43.516707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58858 (local address 127.21.12.131:40139)
0806 06:26:43.517115 (+ 408us) server_negotiation.cc:206] Beginning negotiation
0806 06:26:43.517119 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 06:26:43.517128 (+ 9us) server_negotiation.cc:407] Connection header received
0806 06:26:43.517149 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:26:43.517152 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:26:43.517160 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:26:43.517228 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:26:43.518122 (+ 894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:26:43.518132 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:26:43.518132 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 06:26:43.518136 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:26:43.518169 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 06:26:43.518193 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:26:43.518474 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0806 06:26:43.518497 (+ 23us) server_negotiation.cc:299] Negotiation successful
0806 06:26:43.518509 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":373,"thread_start_us":80,"threads_started":1}
I20260806 06:26:43.518997 28726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:43.517043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33684 (local address 127.21.12.129:46643)
0806 06:26:43.517226 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0806 06:26:43.517230 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 06:26:43.517239 (+ 9us) server_negotiation.cc:407] Connection header received
0806 06:26:43.517261 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:26:43.517263 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:26:43.517272 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:26:43.517511 (+ 239us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:26:43.517843 (+ 332us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:26:43.517848 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:26:43.517849 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 06:26:43.517852 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:26:43.517883 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 06:26:43.517902 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:26:43.518879 (+ 977us) server_negotiation.cc:1035] Waiting for connection context
0806 06:26:43.518903 (+ 24us) server_negotiation.cc:299] Negotiation successful
0806 06:26:43.518919 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":96,"threads_started":1}
I20260806 06:26:43.519037 28425 tablet_service.cc:1467] Processing CreateTablet for tablet fcbcf54c2c944c4080eca0a16fecb750 (DEFAULT_TABLE table=test-table [id=698236c17a0a44259df6572397259d96]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 06:26:43.519343 28425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fcbcf54c2c944c4080eca0a16fecb750. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 06:26:43.519598 28235 tablet_service.cc:1467] Processing CreateTablet for tablet fcbcf54c2c944c4080eca0a16fecb750 (DEFAULT_TABLE table=test-table [id=698236c17a0a44259df6572397259d96]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 06:26:43.519905 28235 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fcbcf54c2c944c4080eca0a16fecb750. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 06:26:43.521808 28727 tablet_bootstrap.cc:492] T fcbcf54c2c944c4080eca0a16fecb750 P 57d6f8f29b3f4792af16b482a70b859f: Bootstrap starting.
I20260806 06:26:43.522218 28612 tablet_service.cc:1467] Processing CreateTablet for tablet fcbcf54c2c944c4080eca0a16fecb750 (DEFAULT_TABLE table=test-table [id=698236c17a0a44259df6572397259d96]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 06:26:43.522346 28731 tablet_bootstrap.cc:492] T fcbcf54c2c944c4080eca0a16fecb750 P 2d5a27fe74d140d593b27586396f6ce6: Bootstrap starting.
I20260806 06:26:43.522557 28612 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fcbcf54c2c944c4080eca0a16fecb750. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 06:26:43.522782 28727 tablet_bootstrap.cc:654] T fcbcf54c2c944c4080eca0a16fecb750 P 57d6f8f29b3f4792af16b482a70b859f: Neither blocks nor log segments found. Creating new log.
I20260806 06:26:43.522964 28731 tablet_bootstrap.cc:654] T fcbcf54c2c944c4080eca0a16fecb750 P 2d5a27fe74d140d593b27586396f6ce6: Neither blocks nor log segments found. Creating new log.
I20260806 06:26:43.523064 28727 log.cc:824] T fcbcf54c2c944c4080eca0a16fecb750 P 57d6f8f29b3f4792af16b482a70b859f: Log is configured to *not* fsync() on all Append() calls
I20260806 06:26:43.523265 28731 log.cc:824] T fcbcf54c2c944c4080eca0a16fecb750 P 2d5a27fe74d140d593b27586396f6ce6: Log is configured to *not* fsync() on all Append() calls
I20260806 06:26:43.523720 28727 tablet_bootstrap.cc:492] T fcbcf54c2c944c4080eca0a16fecb750 P 57d6f8f29b3f4792af16b482a70b859f: No bootstrap required, opened a new log
I20260806 06:26:43.523773 28727 ts_tablet_manager.cc:1397] T fcbcf54c2c944c4080eca0a16fecb750 P 57d6f8f29b3f4792af16b482a70b859f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260806 06:26:43.525156 28735 tablet_bootstrap.cc:492] T fcbcf54c2c944c4080eca0a16fecb750 P 5e22e0ce55394248a1d9f866db8258f7: Bootstrap starting.
I20260806 06:26:43.525811 28735 tablet_bootstrap.cc:654] T fcbcf54c2c944c4080eca0a16fecb750 P 5e22e0ce55394248a1d9f866db8258f7: Neither blocks nor log segments found. Creating new log.
I20260806 06:26:43.525923 28727 raft_consensus.cc:348] T fcbcf54c2c944c4080eca0a16fecb750 P 57d6f8f29b3f4792af16b482a70b859f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e22e0ce55394248a1d9f866db8258f7" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 40139 } } peers { permanent_uuid: "2d5a27fe74d140d593b27586396f6ce6" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 46643 } } peers { permanent_uuid: "57d6f8f29b3f4792af16b482a70b859f" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 41347 } }
I20260806 06:26:43.526089 28727 raft_consensus.cc:374] T fcbcf54c2c944c4080eca0a16fecb750 P 57d6f8f29b3f4792af16b482a70b859f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 06:26:43.526122 28735 log.cc:824] T fcbcf54c2c944c4080eca0a16fecb750 P 5e22e0ce55394248a1d9f866db8258f7: Log is configured to *not* fsync() on all Append() calls
I20260806 06:26:43.526135 28727 raft_consensus.cc:729] T fcbcf54c2c944c4080eca0a16fecb750 P 57d6f8f29b3f4792af16b482a70b859f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57d6f8f29b3f4792af16b482a70b859f, State: Initialized, Role: FOLLOWER
I20260806 06:26:43.526253 28727 consensus_queue.cc:260] T fcbcf54c2c944c4080eca0a16fecb750 P 57d6f8f29b3f4792af16b482a70b859f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e22e0ce55394248a1d9f866db8258f7" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 40139 } } peers { permanent_uuid: "2d5a27fe74d140d593b27586396f6ce6" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 46643 } } peers { permanent_uuid: "57d6f8f29b3f4792af16b482a70b859f" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 41347 } }
I20260806 06:26:43.526544 28727 ts_tablet_manager.cc:1428] T fcbcf54c2c944c4080eca0a16fecb750 P 57d6f8f29b3f4792af16b482a70b859f: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260806 06:26:43.526670 28522 heartbeater.cc:503] Master 127.21.12.190:37381 was elected leader, sending a full tablet report...
I20260806 06:26:43.526896 28731 tablet_bootstrap.cc:492] T fcbcf54c2c944c4080eca0a16fecb750 P 2d5a27fe74d140d593b27586396f6ce6: No bootstrap required, opened a new log
I20260806 06:26:43.526969 28731 ts_tablet_manager.cc:1397] T fcbcf54c2c944c4080eca0a16fecb750 P 2d5a27fe74d140d593b27586396f6ce6: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260806 06:26:43.528955 28731 raft_consensus.cc:348] T fcbcf54c2c944c4080eca0a16fecb750 P 2d5a27fe74d140d593b27586396f6ce6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e22e0ce55394248a1d9f866db8258f7" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 40139 } } peers { permanent_uuid: "2d5a27fe74d140d593b27586396f6ce6" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 46643 } } peers { permanent_uuid: "57d6f8f29b3f4792af16b482a70b859f" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 41347 } }
I20260806 06:26:43.529085 28731 raft_consensus.cc:374] T fcbcf54c2c944c4080eca0a16fecb750 P 2d5a27fe74d140d593b27586396f6ce6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 06:26:43.529105 28731 raft_consensus.cc:729] T fcbcf54c2c944c4080eca0a16fecb750 P 2d5a27fe74d140d593b27586396f6ce6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d5a27fe74d140d593b27586396f6ce6, State: Initialized, Role: FOLLOWER
I20260806 06:26:43.529186 28731 consensus_queue.cc:260] T fcbcf54c2c944c4080eca0a16fecb750 P 2d5a27fe74d140d593b27586396f6ce6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e22e0ce55394248a1d9f866db8258f7" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 40139 } } peers { permanent_uuid: "2d5a27fe74d140d593b27586396f6ce6" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 46643 } } peers { permanent_uuid: "57d6f8f29b3f4792af16b482a70b859f" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 41347 } }
I20260806 06:26:43.529368 28731 ts_tablet_manager.cc:1428] T fcbcf54c2c944c4080eca0a16fecb750 P 2d5a27fe74d140d593b27586396f6ce6: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260806 06:26:43.529429 28334 heartbeater.cc:503] Master 127.21.12.190:37381 was elected leader, sending a full tablet report...
I20260806 06:26:43.529745 28735 tablet_bootstrap.cc:492] T fcbcf54c2c944c4080eca0a16fecb750 P 5e22e0ce55394248a1d9f866db8258f7: No bootstrap required, opened a new log
I20260806 06:26:43.529799 28735 ts_tablet_manager.cc:1397] T fcbcf54c2c944c4080eca0a16fecb750 P 5e22e0ce55394248a1d9f866db8258f7: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.001s
I20260806 06:26:43.531692 28735 raft_consensus.cc:348] T fcbcf54c2c944c4080eca0a16fecb750 P 5e22e0ce55394248a1d9f866db8258f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e22e0ce55394248a1d9f866db8258f7" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 40139 } } peers { permanent_uuid: "2d5a27fe74d140d593b27586396f6ce6" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 46643 } } peers { permanent_uuid: "57d6f8f29b3f4792af16b482a70b859f" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 41347 } }
I20260806 06:26:43.531805 28735 raft_consensus.cc:374] T fcbcf54c2c944c4080eca0a16fecb750 P 5e22e0ce55394248a1d9f866db8258f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 06:26:43.531826 28735 raft_consensus.cc:729] T fcbcf54c2c944c4080eca0a16fecb750 P 5e22e0ce55394248a1d9f866db8258f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e22e0ce55394248a1d9f866db8258f7, State: Initialized, Role: FOLLOWER
I20260806 06:26:43.531929 28735 consensus_queue.cc:260] T fcbcf54c2c944c4080eca0a16fecb750 P 5e22e0ce55394248a1d9f866db8258f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e22e0ce55394248a1d9f866db8258f7" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 40139 } } peers { permanent_uuid: "2d5a27fe74d140d593b27586396f6ce6" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 46643 } } peers { permanent_uuid: "57d6f8f29b3f4792af16b482a70b859f" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 41347 } }
I20260806 06:26:43.532152 28735 ts_tablet_manager.cc:1428] T fcbcf54c2c944c4080eca0a16fecb750 P 5e22e0ce55394248a1d9f866db8258f7: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20260806 06:26:43.532214 28709 heartbeater.cc:503] Master 127.21.12.190:37381 was elected leader, sending a full tablet report...
W20260806 06:26:43.535269 28336 tablet.cc:2369] T fcbcf54c2c944c4080eca0a16fecb750 P 2d5a27fe74d140d593b27586396f6ce6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 06:26:43.571252 28740 raft_consensus.cc:482] T fcbcf54c2c944c4080eca0a16fecb750 P 5e22e0ce55394248a1d9f866db8258f7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 06:26:43.571379 28740 raft_consensus.cc:504] T fcbcf54c2c944c4080eca0a16fecb750 P 5e22e0ce55394248a1d9f866db8258f7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e22e0ce55394248a1d9f866db8258f7" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 40139 } } peers { permanent_uuid: "2d5a27fe74d140d593b27586396f6ce6" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 46643 } } peers { permanent_uuid: "57d6f8f29b3f4792af16b482a70b859f" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 41347 } }
I20260806 06:26:43.571749 28740 leader_election.cc:288] T fcbcf54c2c944c4080eca0a16fecb750 P 5e22e0ce55394248a1d9f866db8258f7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2d5a27fe74d140d593b27586396f6ce6 (127.21.12.129:46643), 57d6f8f29b3f4792af16b482a70b859f (127.21.12.130:41347)
I20260806 06:26:43.572983 28726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:43.572006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.131:39313 (local address 127.21.12.129:46643)
0806 06:26:43.572052 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0806 06:26:43.572054 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:26:43.572061 (+ 7us) server_negotiation.cc:407] Connection header received
0806 06:26:43.572087 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:26:43.572088 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:26:43.572093 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:26:43.572151 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:26:43.572758 (+ 607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:26:43.572759 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:26:43.572759 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 06:26:43.572763 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:26:43.572780 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 06:26:43.572784 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:26:43.572835 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0806 06:26:43.572903 (+ 68us) server_negotiation.cc:299] Negotiation successful
0806 06:26:43.572916 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260806 06:26:43.573002 28713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:43.571922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.129:46643 (local address 127.21.12.131:39313)
0806 06:26:43.571973 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0806 06:26:43.571984 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0806 06:26:43.572023 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:26:43.572167 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:26:43.572168 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:26:43.572171 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0806 06:26:43.572651 (+ 480us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 06:26:43.572652 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0806 06:26:43.572659 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 06:26:43.572672 (+ 13us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 06:26:43.572850 (+ 178us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 06:26:43.572851 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 06:26:43.572852 (+ 1us) client_negotiation.cc:769] Sending connection context
0806 06:26:43.572898 (+ 46us) client_negotiation.cc:240] Negotiation successful
0806 06:26:43.572913 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260806 06:26:43.573019 28723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:43.572003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.131:40207 (local address 127.21.12.130:41347)
0806 06:26:43.572060 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0806 06:26:43.572062 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:26:43.572342 (+ 280us) server_negotiation.cc:407] Connection header received
0806 06:26:43.572373 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:26:43.572374 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:26:43.572380 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:26:43.572435 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:26:43.572821 (+ 386us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:26:43.572822 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:26:43.572823 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 06:26:43.572827 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:26:43.572842 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 06:26:43.572846 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:26:43.572894 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0806 06:26:43.572937 (+ 43us) server_negotiation.cc:299] Negotiation successful
0806 06:26:43.572951 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260806 06:26:43.573084 28741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:43.571925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.130:41347 (local address 127.21.12.131:40207)
0806 06:26:43.572268 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0806 06:26:43.572278 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0806 06:26:43.572330 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:26:43.572440 (+ 110us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:26:43.572441 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:26:43.572442 (+ 1us) client_negotiation.cc:189] Negotiated authn=SASL
0806 06:26:43.572742 (+ 300us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 06:26:43.572743 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0806 06:26:43.572748 (+ 5us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 06:26:43.572756 (+ 8us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 06:26:43.572896 (+ 140us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 06:26:43.572897 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 06:26:43.572898 (+ 1us) client_negotiation.cc:769] Sending connection context
0806 06:26:43.572940 (+ 42us) client_negotiation.cc:240] Negotiation successful
0806 06:26:43.572951 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":319,"mutex_wait_us":143,"thread_start_us":126,"threads_started":1}
I20260806 06:26:43.573338 28270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcbcf54c2c944c4080eca0a16fecb750" candidate_uuid: "5e22e0ce55394248a1d9f866db8258f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d5a27fe74d140d593b27586396f6ce6" is_pre_election: true
I20260806 06:26:43.573345 28456 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcbcf54c2c944c4080eca0a16fecb750" candidate_uuid: "5e22e0ce55394248a1d9f866db8258f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57d6f8f29b3f4792af16b482a70b859f" is_pre_election: true
I20260806 06:26:43.573480 28270 raft_consensus.cc:2454] T fcbcf54c2c944c4080eca0a16fecb750 P 2d5a27fe74d140d593b27586396f6ce6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e22e0ce55394248a1d9f866db8258f7 in term 0.
I20260806 06:26:43.573482 28456 raft_consensus.cc:2454] T fcbcf54c2c944c4080eca0a16fecb750 P 57d6f8f29b3f4792af16b482a70b859f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e22e0ce55394248a1d9f866db8258f7 in term 0.
I20260806 06:26:43.573699 28556 leader_election.cc:302] T fcbcf54c2c944c4080eca0a16fecb750 P 5e22e0ce55394248a1d9f866db8258f7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 57d6f8f29b3f4792af16b482a70b859f, 5e22e0ce55394248a1d9f866db8258f7; no voters:
I20260806 06:26:43.573819 28740 raft_consensus.cc:2781] T fcbcf54c2c944c4080eca0a16fecb750 P 5e22e0ce55394248a1d9f866db8258f7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 06:26:43.573876 28740 raft_consensus.cc:482] T fcbcf54c2c944c4080eca0a16fecb750 P 5e22e0ce55394248a1d9f866db8258f7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 06:26:43.573899 28740 raft_consensus.cc:3037] T fcbcf54c2c944c4080eca0a16fecb750 P 5e22e0ce55394248a1d9f866db8258f7 [term 0 FOLLOWER]: Advancing to term 1
I20260806 06:26:43.574626 28740 raft_consensus.cc:504] T fcbcf54c2c944c4080eca0a16fecb750 P 5e22e0ce55394248a1d9f866db8258f7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e22e0ce55394248a1d9f866db8258f7" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 40139 } } peers { permanent_uuid: "2d5a27fe74d140d593b27586396f6ce6" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 46643 } } peers { permanent_uuid: "57d6f8f29b3f4792af16b482a70b859f" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 41347 } }
I20260806 06:26:43.574738 28740 leader_election.cc:288] T fcbcf54c2c944c4080eca0a16fecb750 P 5e22e0ce55394248a1d9f866db8258f7 [CANDIDATE]: Term 1 election: Requested vote from peers 2d5a27fe74d140d593b27586396f6ce6 (127.21.12.129:46643), 57d6f8f29b3f4792af16b482a70b859f (127.21.12.130:41347)
I20260806 06:26:43.574884 28270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcbcf54c2c944c4080eca0a16fecb750" candidate_uuid: "5e22e0ce55394248a1d9f866db8258f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d5a27fe74d140d593b27586396f6ce6"
I20260806 06:26:43.574895 28456 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcbcf54c2c944c4080eca0a16fecb750" candidate_uuid: "5e22e0ce55394248a1d9f866db8258f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57d6f8f29b3f4792af16b482a70b859f"
I20260806 06:26:43.574955 28456 raft_consensus.cc:3037] T fcbcf54c2c944c4080eca0a16fecb750 P 57d6f8f29b3f4792af16b482a70b859f [term 0 FOLLOWER]: Advancing to term 1
I20260806 06:26:43.574958 28270 raft_consensus.cc:3037] T fcbcf54c2c944c4080eca0a16fecb750 P 2d5a27fe74d140d593b27586396f6ce6 [term 0 FOLLOWER]: Advancing to term 1
I20260806 06:26:43.575605 28270 raft_consensus.cc:2454] T fcbcf54c2c944c4080eca0a16fecb750 P 2d5a27fe74d140d593b27586396f6ce6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e22e0ce55394248a1d9f866db8258f7 in term 1.
I20260806 06:26:43.575677 28456 raft_consensus.cc:2454] T fcbcf54c2c944c4080eca0a16fecb750 P 57d6f8f29b3f4792af16b482a70b859f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e22e0ce55394248a1d9f866db8258f7 in term 1.
I20260806 06:26:43.575793 28552 leader_election.cc:302] T fcbcf54c2c944c4080eca0a16fecb750 P 5e22e0ce55394248a1d9f866db8258f7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d5a27fe74d140d593b27586396f6ce6, 5e22e0ce55394248a1d9f866db8258f7; no voters:
I20260806 06:26:43.575897 28740 raft_consensus.cc:2781] T fcbcf54c2c944c4080eca0a16fecb750 P 5e22e0ce55394248a1d9f866db8258f7 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 06:26:43.576081 28740 raft_consensus.cc:686] T fcbcf54c2c944c4080eca0a16fecb750 P 5e22e0ce55394248a1d9f866db8258f7 [term 1 LEADER]: Becoming Leader. State: Replica: 5e22e0ce55394248a1d9f866db8258f7, State: Running, Role: LEADER
I20260806 06:26:43.576166 28740 consensus_queue.cc:237] T fcbcf54c2c944c4080eca0a16fecb750 P 5e22e0ce55394248a1d9f866db8258f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e22e0ce55394248a1d9f866db8258f7" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 40139 } } peers { permanent_uuid: "2d5a27fe74d140d593b27586396f6ce6" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 46643 } } peers { permanent_uuid: "57d6f8f29b3f4792af16b482a70b859f" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 41347 } }
I20260806 06:26:43.577023 28059 catalog_manager.cc:5495] T fcbcf54c2c944c4080eca0a16fecb750 P 5e22e0ce55394248a1d9f866db8258f7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5e22e0ce55394248a1d9f866db8258f7 (127.21.12.131). New cstate: current_term: 1 leader_uuid: "5e22e0ce55394248a1d9f866db8258f7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e22e0ce55394248a1d9f866db8258f7" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 40139 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2d5a27fe74d140d593b27586396f6ce6" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 46643 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "57d6f8f29b3f4792af16b482a70b859f" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 41347 } health_report { overall_health: UNKNOWN } } }
I20260806 06:26:43.594043 28721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:43.593195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58868 (local address 127.21.12.131:40139)
0806 06:26:43.593252 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0806 06:26:43.593254 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:26:43.593263 (+ 9us) server_negotiation.cc:407] Connection header received
0806 06:26:43.593298 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:26:43.593299 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:26:43.593311 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:26:43.593360 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:26:43.593832 (+ 472us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:26:43.593833 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:26:43.593833 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 06:26:43.593837 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:26:43.593851 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 06:26:43.593855 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:26:43.593902 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0806 06:26:43.593959 (+ 57us) server_negotiation.cc:299] Negotiation successful
0806 06:26:43.593972 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260806 06:26:43.595734 28456 raft_consensus.cc:1261] T fcbcf54c2c944c4080eca0a16fecb750 P 57d6f8f29b3f4792af16b482a70b859f [term 1 FOLLOWER]: Refusing update from remote peer 5e22e0ce55394248a1d9f866db8258f7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 06:26:43.595734 28270 raft_consensus.cc:1261] T fcbcf54c2c944c4080eca0a16fecb750 P 2d5a27fe74d140d593b27586396f6ce6 [term 1 FOLLOWER]: Refusing update from remote peer 5e22e0ce55394248a1d9f866db8258f7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 06:26:43.596069 28742 consensus_queue.cc:1035] T fcbcf54c2c944c4080eca0a16fecb750 P 5e22e0ce55394248a1d9f866db8258f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "57d6f8f29b3f4792af16b482a70b859f" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 41347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 06:26:43.596180 28740 consensus_queue.cc:1035] T fcbcf54c2c944c4080eca0a16fecb750 P 5e22e0ce55394248a1d9f866db8258f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d5a27fe74d140d593b27586396f6ce6" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 46643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 06:26:43.598148 28749 mvcc.cc:205] Tried to move back new op lower bound from 7315446184325767168 to 7315446184248717312. Current Snapshot: MvccSnapshot[applied={T|T < 7315446184325767168}]
I20260806 06:26:43.599713 28752 mvcc.cc:205] Tried to move back new op lower bound from 7315446184325767168 to 7315446184248717312. Current Snapshot: MvccSnapshot[applied={T|T < 7315446184325767168}]
I20260806 06:26:43.600402 28061 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54884:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 06:26:43.600486 28061 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54884:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 06:26:43.601122 28061 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4cef0803e77b40bf8407c0022af7698a: Sending DeleteTablet for 3 replicas of tablet fcbcf54c2c944c4080eca0a16fecb750
I20260806 06:26:43.601424 28425 tablet_service.cc:1514] Processing DeleteTablet for tablet fcbcf54c2c944c4080eca0a16fecb750 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 06:26:43 UTC) from {username='slave'} at 127.0.0.1:60996
I20260806 06:26:43.601487 28235 tablet_service.cc:1514] Processing DeleteTablet for tablet fcbcf54c2c944c4080eca0a16fecb750 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 06:26:43 UTC) from {username='slave'} at 127.0.0.1:33684
I20260806 06:26:43.601533 28612 tablet_service.cc:1514] Processing DeleteTablet for tablet fcbcf54c2c944c4080eca0a16fecb750 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 06:26:43 UTC) from {username='slave'} at 127.0.0.1:58858
I20260806 06:26:43.601660 28762 tablet_replica.cc:330] T fcbcf54c2c944c4080eca0a16fecb750 P 57d6f8f29b3f4792af16b482a70b859f: stopping tablet replica
I20260806 06:26:43.601758 28762 raft_consensus.cc:2229] T fcbcf54c2c944c4080eca0a16fecb750 P 57d6f8f29b3f4792af16b482a70b859f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 06:26:43.601781 28764 tablet_replica.cc:330] T fcbcf54c2c944c4080eca0a16fecb750 P 2d5a27fe74d140d593b27586396f6ce6: stopping tablet replica
I20260806 06:26:43.601858 28762 raft_consensus.cc:2258] T fcbcf54c2c944c4080eca0a16fecb750 P 57d6f8f29b3f4792af16b482a70b859f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 06:26:43.601897 21554 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHhLhT/build/release/bin/kudu with pid 28152
I20260806 06:26:43.602020 28764 raft_consensus.cc:2229] T fcbcf54c2c944c4080eca0a16fecb750 P 2d5a27fe74d140d593b27586396f6ce6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 06:26:43.609881 28765 tablet_replica.cc:330] T fcbcf54c2c944c4080eca0a16fecb750 P 5e22e0ce55394248a1d9f866db8258f7: stopping tablet replica
I20260806 06:26:43.610077 28765 raft_consensus.cc:2229] T fcbcf54c2c944c4080eca0a16fecb750 P 5e22e0ce55394248a1d9f866db8258f7 [term 1 LEADER]: Raft consensus shutting down.
I20260806 06:26:43.610164 28762 ts_tablet_manager.cc:1905] T fcbcf54c2c944c4080eca0a16fecb750 P 57d6f8f29b3f4792af16b482a70b859f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 06:26:43.610270 28765 raft_consensus.cc:2258] T fcbcf54c2c944c4080eca0a16fecb750 P 5e22e0ce55394248a1d9f866db8258f7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 06:26:43.610782 28765 ts_tablet_manager.cc:1905] T fcbcf54c2c944c4080eca0a16fecb750 P 5e22e0ce55394248a1d9f866db8258f7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 06:26:43.611959 28762 ts_tablet_manager.cc:1918] T fcbcf54c2c944c4080eca0a16fecb750 P 57d6f8f29b3f4792af16b482a70b859f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 06:26:43.612046 28762 log.cc:1196] T fcbcf54c2c944c4080eca0a16fecb750 P 57d6f8f29b3f4792af16b482a70b859f: Deleting WAL directory at /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785997581049705-21554-0/minicluster-data/ts-1/wal/wals/fcbcf54c2c944c4080eca0a16fecb750
I20260806 06:26:43.612303 28762 ts_tablet_manager.cc:1939] T fcbcf54c2c944c4080eca0a16fecb750 P 57d6f8f29b3f4792af16b482a70b859f: Deleting consensus metadata
I20260806 06:26:43.612586 28765 ts_tablet_manager.cc:1918] T fcbcf54c2c944c4080eca0a16fecb750 P 5e22e0ce55394248a1d9f866db8258f7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 06:26:43.612712 28765 log.cc:1196] T fcbcf54c2c944c4080eca0a16fecb750 P 5e22e0ce55394248a1d9f866db8258f7: Deleting WAL directory at /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785997581049705-21554-0/minicluster-data/ts-2/wal/wals/fcbcf54c2c944c4080eca0a16fecb750
I20260806 06:26:43.612712 28046 catalog_manager.cc:4841] TS 57d6f8f29b3f4792af16b482a70b859f (127.21.12.130:41347): tablet fcbcf54c2c944c4080eca0a16fecb750 (table test-table [id=698236c17a0a44259df6572397259d96]) successfully deleted
I20260806 06:26:43.613045 28765 ts_tablet_manager.cc:1939] T fcbcf54c2c944c4080eca0a16fecb750 P 5e22e0ce55394248a1d9f866db8258f7: Deleting consensus metadata
I20260806 06:26:43.613440 28042 catalog_manager.cc:4841] TS 5e22e0ce55394248a1d9f866db8258f7 (127.21.12.131:40139): tablet fcbcf54c2c944c4080eca0a16fecb750 (table test-table [id=698236c17a0a44259df6572397259d96]) successfully deleted
W20260806 06:26:43.614215 28042 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.12.129:46643 (error 108)
I20260806 06:26:43.614392 21554 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHhLhT/build/release/bin/kudu with pid 28338
I20260806 06:26:43.627763 28724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:43.627572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.129:46643 (local address 127.0.0.1:33694)
0806 06:26:43.627640 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0806 06:26:43.627714 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.129:46643: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260806 06:26:43.627925 28042 catalog_manager.cc:4584] TS 2d5a27fe74d140d593b27586396f6ce6 (127.21.12.129:46643): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet fcbcf54c2c944c4080eca0a16fecb750: Network error: Client connection negotiation failed: client connection to 127.21.12.129:46643: connect: Connection refused (error 111)
I20260806 06:26:43.630964 21554 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHhLhT/build/release/bin/kudu with pid 28525
I20260806 06:26:43.639851 21554 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHhLhT/build/release/bin/kudu with pid 28019
2026-08-06T06:26:43Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (620 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260806 06:26:43.662237 21554 mini_oidc.cc:152] Starting JWKS server
I20260806 06:26:43.663440 21554 webserver.cc:460] Webserver started at https://127.0.0.1:38657/ using document root <none> and password file <none>
I20260806 06:26:43.663532 21554 mini_oidc.cc:189] Starting OIDC Discovery server
I20260806 06:26:43.663703 21554 webserver.cc:460] Webserver started at http://127.0.0.1:42881/ using document root <none> and password file <none>
2026-08-06T06:26:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T06:26:43Z Disabled control of system clock
I20260806 06:26:43.671623 21554 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHhLhT/build/release/bin/kudu
/tmp/dist-test-taskFHhLhT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785997581049705-21554-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785997581049705-21554-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785997581049705-21554-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785997581049705-21554-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.12.190:44891
--webserver_interface=127.21.12.190
--webserver_port=0
--builtin_ntp_servers=127.21.12.148:40967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.12.190:44891
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38657/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785997581049705-21554-0/testchainca.cert with env {}
W20260806 06:26:43.750888 28776 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 06:26:43.751085 28776 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 06:26:43.751106 28776 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 06:26:43.752216 28776 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 06:26:43.752250 28776 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 06:26:43.752261 28776 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 06:26:43.752272 28776 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 06:26:43.752283 28776 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260806 06:26:43.752293 28776 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260806 06:26:43.752303 28776 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:38657/jwks.json
I20260806 06:26:43.753877 28776 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.148:40967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785997581049705-21554-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785997581049705-21554-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.12.190:44891
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.12.190:44891
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38657/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785997581049705-21554-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.12.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785997581049705-21554-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28776
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785997581049705-21554-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 05:11:02 UTC on ccycloud.quasar-hvzbbz.root.comops.site
build id 4569617
I20260806 06:26:43.754074 28776 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 06:26:43.754237 28776 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 06:26:43.758802 28787 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:26:43.758826 28784 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:26:43.758863 28776 server_base.cc:1034] running on GCE node
W20260806 06:26:43.758839 28783 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:26:43.759245 28776 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 06:26:43.759519 28776 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 06:26:43.760663 28776 hybrid_clock.cc:648] HybridClock initialized: now 1785997603760646 us; error 29 us; skew 500 ppm
I20260806 06:26:43.761782 28776 webserver.cc:460] Webserver started at http://127.21.12.190:45693/ using document root <none> and password file <none>
I20260806 06:26:43.761993 28776 fs_manager.cc:359] Metadata directory not provided
I20260806 06:26:43.762040 28776 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 06:26:43.762172 28776 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 06:26:43.763235 28776 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785997581049705-21554-0/minicluster-data/master-0/data/instance:
uuid: "f2fb2fe1121c44ba99fb4979a3c7c991"
format_stamp: "Formatted at 2026-08-06 06:26:43 on dist-test-slave-4tdj"
server_key: "5b2d1cb9ce5eb8d37dd30c356dbb6260"
server_key_iv: "8e4dabad133a01f80465dd16867dc734"
server_key_version: "encryptionkey@0"
I20260806 06:26:43.763564 28776 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785997581049705-21554-0/minicluster-data/master-0/wal/instance:
uuid: "f2fb2fe1121c44ba99fb4979a3c7c991"
format_stamp: "Formatted at 2026-08-06 06:26:43 on dist-test-slave-4tdj"
server_key: "5b2d1cb9ce5eb8d37dd30c356dbb6260"
server_key_iv: "8e4dabad133a01f80465dd16867dc734"
server_key_version: "encryptionkey@0"
I20260806 06:26:43.764744 28776 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260806 06:26:43.765494 28795 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 06:26:43.765641 28776 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260806 06:26:43.765724 28776 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785997581049705-21554-0/minicluster-data/master-0/data,/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785997581049705-21554-0/minicluster-data/master-0/wal
uuid: "f2fb2fe1121c44ba99fb4979a3c7c991"
format_stamp: "Formatted at 2026-08-06 06:26:43 on dist-test-slave-4tdj"
server_key: "5b2d1cb9ce5eb8d37dd30c356dbb6260"
server_key_iv: "8e4dabad133a01f80465dd16867dc734"
server_key_version: "encryptionkey@0"
I20260806 06:26:43.765789 28776 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785997581049705-21554-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785997581049705-21554-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785997581049705-21554-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 06:26:43.781689 28776 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 06:26:43.781976 28776 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 06:26:43.782106 28776 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 06:26:43.785843 28776 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.190:44891
I20260806 06:26:43.785881 28870 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.190:44891 every 8 connection(s)
I20260806 06:26:43.786199 28776 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785997581049705-21554-0/minicluster-data/master-0/data/info.pb
I20260806 06:26:43.787214 28871 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 06:26:43.789448 28871 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f2fb2fe1121c44ba99fb4979a3c7c991: Bootstrap starting.
I20260806 06:26:43.790004 28871 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f2fb2fe1121c44ba99fb4979a3c7c991: Neither blocks nor log segments found. Creating new log.
I20260806 06:26:43.790278 28871 log.cc:824] T 00000000000000000000000000000000 P f2fb2fe1121c44ba99fb4979a3c7c991: Log is configured to *not* fsync() on all Append() calls
I20260806 06:26:43.790902 28871 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f2fb2fe1121c44ba99fb4979a3c7c991: No bootstrap required, opened a new log
I20260806 06:26:43.792397 28871 raft_consensus.cc:348] T 00000000000000000000000000000000 P f2fb2fe1121c44ba99fb4979a3c7c991 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2fb2fe1121c44ba99fb4979a3c7c991" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 44891 } }
I20260806 06:26:43.792527 28871 raft_consensus.cc:374] T 00000000000000000000000000000000 P f2fb2fe1121c44ba99fb4979a3c7c991 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 06:26:43.792548 28871 raft_consensus.cc:729] T 00000000000000000000000000000000 P f2fb2fe1121c44ba99fb4979a3c7c991 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2fb2fe1121c44ba99fb4979a3c7c991, State: Initialized, Role: FOLLOWER
I20260806 06:26:43.792687 28871 consensus_queue.cc:260] T 00000000000000000000000000000000 P f2fb2fe1121c44ba99fb4979a3c7c991 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2fb2fe1121c44ba99fb4979a3c7c991" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 44891 } }
I20260806 06:26:43.792748 28871 raft_consensus.cc:388] T 00000000000000000000000000000000 P f2fb2fe1121c44ba99fb4979a3c7c991 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 06:26:43.792768 28871 raft_consensus.cc:482] T 00000000000000000000000000000000 P f2fb2fe1121c44ba99fb4979a3c7c991 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 06:26:43.792811 28871 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f2fb2fe1121c44ba99fb4979a3c7c991 [term 0 FOLLOWER]: Advancing to term 1
I20260806 06:26:43.793303 28871 raft_consensus.cc:504] T 00000000000000000000000000000000 P f2fb2fe1121c44ba99fb4979a3c7c991 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2fb2fe1121c44ba99fb4979a3c7c991" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 44891 } }
I20260806 06:26:43.793388 28871 leader_election.cc:302] T 00000000000000000000000000000000 P f2fb2fe1121c44ba99fb4979a3c7c991 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f2fb2fe1121c44ba99fb4979a3c7c991; no voters:
I20260806 06:26:43.793533 28871 leader_election.cc:288] T 00000000000000000000000000000000 P f2fb2fe1121c44ba99fb4979a3c7c991 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 06:26:43.793610 28874 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f2fb2fe1121c44ba99fb4979a3c7c991 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 06:26:43.793754 28871 sys_catalog.cc:564] T 00000000000000000000000000000000 P f2fb2fe1121c44ba99fb4979a3c7c991 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 06:26:43.793772 28874 raft_consensus.cc:686] T 00000000000000000000000000000000 P f2fb2fe1121c44ba99fb4979a3c7c991 [term 1 LEADER]: Becoming Leader. State: Replica: f2fb2fe1121c44ba99fb4979a3c7c991, State: Running, Role: LEADER
I20260806 06:26:43.793888 28874 consensus_queue.cc:237] T 00000000000000000000000000000000 P f2fb2fe1121c44ba99fb4979a3c7c991 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2fb2fe1121c44ba99fb4979a3c7c991" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 44891 } }
I20260806 06:26:43.794252 28876 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2fb2fe1121c44ba99fb4979a3c7c991 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f2fb2fe1121c44ba99fb4979a3c7c991. Latest consensus state: current_term: 1 leader_uuid: "f2fb2fe1121c44ba99fb4979a3c7c991" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2fb2fe1121c44ba99fb4979a3c7c991" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 44891 } } }
I20260806 06:26:43.794353 28876 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2fb2fe1121c44ba99fb4979a3c7c991 [sys.catalog]: This master's current role is: LEADER
I20260806 06:26:43.794348 28875 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2fb2fe1121c44ba99fb4979a3c7c991 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f2fb2fe1121c44ba99fb4979a3c7c991" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2fb2fe1121c44ba99fb4979a3c7c991" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 44891 } } }
I20260806 06:26:43.794651 28875 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2fb2fe1121c44ba99fb4979a3c7c991 [sys.catalog]: This master's current role is: LEADER
I20260806 06:26:43.794696 28883 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 06:26:43.795197 28883 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 06:26:43.795722 21554 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHhLhT/build/release/bin/kudu as pid 28776
I20260806 06:26:43.795817 21554 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785997581049705-21554-0/minicluster-data/master-0/wal/instance
I20260806 06:26:43.796017 21554 external_mini_cluster.cc:1428] Setting key 71073693e47492f957f9261f4791484a
I20260806 06:26:43.796792 28883 catalog_manager.cc:1347] Generated new cluster ID: 9abf7b7114ad466da472d70eaea42d87
I20260806 06:26:43.796845 28883 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 06:26:43.802332 28901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:43.796645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57754 (local address 127.21.12.190:44891)
0806 06:26:43.798428 (+ 1783us) server_negotiation.cc:206] Beginning negotiation
0806 06:26:43.798435 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 06:26:43.798459 (+ 24us) server_negotiation.cc:407] Connection header received
0806 06:26:43.798567 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:26:43.798571 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:26:43.798622 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:26:43.798741 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:26:43.798931 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:43.799807 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:43.800572 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:43.801000 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:43.801114 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:43.801726 (+ 612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:26:43.801730 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:26:43.801731 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 06:26:43.801897 (+ 166us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:26:43.801930 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 06:26:43.801941 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:26:43.801995 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0806 06:26:43.802071 (+ 76us) server_negotiation.cc:299] Negotiation successful
0806 06:26:43.802146 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1715,"thread_start_us":74,"threads_started":1}
I20260806 06:26:43.803160 21554 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 06:26:43.810829 28883 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 06:26:43.811471 28883 catalog_manager.cc:1504] Loading token signing keys...
I20260806 06:26:43.821362 28883 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f2fb2fe1121c44ba99fb4979a3c7c991: Generated new TSK 0
I20260806 06:26:43.821547 28883 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260806 06:26:43.842133 28910 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260806 06:26:43.842319 28910 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260806 06:26:43.842573 28901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:43.827765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57762 (local address 127.21.12.190:44891)
0806 06:26:43.827813 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0806 06:26:43.827815 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:26:43.827945 (+ 130us) server_negotiation.cc:407] Connection header received
0806 06:26:43.828042 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:26:43.828043 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:26:43.828055 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:26:43.828108 (+ 53us) server_negotiation.cc:226] Negotiated authn=JWT
0806 06:26:43.828233 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:43.828722 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:43.829464 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:43.829659 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:43.829711 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:43.829901 (+ 190us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0806 06:26:43.842363 (+ 12462us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38657/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0806 06:26:43.842522 (+ 159us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:57762: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38657/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":24}
W20260806 06:26:43.842628 28901 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:57762: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38657/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260806 06:26:43.843009 28909 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 06:26:43.827719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:44891 (local address 127.0.0.1:57762)
0806 06:26:43.827883 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0806 06:26:43.827893 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0806 06:26:43.827956 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:26:43.828111 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:26:43.828112 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:26:43.828114 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0806 06:26:43.828176 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:43.828178 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:43.828800 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:43.828801 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:43.829388 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:43.829390 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:43.829720 (+ 330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:43.829721 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:43.829787 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:43.829807 (+ 20us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0806 06:26:43.842535 (+ 12728us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38657/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0806 06:26:43.842667 (+ 132us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.21.12.190:44891: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38657/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":81,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.21.12.190:44891: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38657/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260806 06:26:43.843426 21554 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHhLhT/build/release/bin/kudu with pid 28776
2026-08-06T06:26:43Z chronyd exiting
I20260806 06:26:43.890354 21554 test_util.cc:175] -----------------------------------------------
I20260806 06:26:43.890437 21554 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785997581049705-21554-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (229 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260806 06:26:43.891179 21554 mini_oidc.cc:152] Starting JWKS server
I20260806 06:26:43.891822 21554 webserver.cc:460] Webserver started at https://127.0.0.1:41543/ using document root <none> and password file <none>
I20260806 06:26:43.891861 21554 mini_oidc.cc:189] Starting OIDC Discovery server
I20260806 06:26:43.891968 21554 webserver.cc:460] Webserver started at http://127.0.0.1:37485/ using document root <none> and password file <none>
2026-08-06T06:26:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T06:26:43Z Disabled control of system clock
I20260806 06:26:43.899971 21554 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHhLhT/build/release/bin/kudu
/tmp/dist-test-taskFHhLhT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785997581049705-21554-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785997581049705-21554-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785997581049705-21554-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785997581049705-21554-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.12.190:41293
--webserver_interface=127.21.12.190
--webserver_port=0
--builtin_ntp_servers=127.21.12.148:45907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.12.190:41293
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41543/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785997581049705-21554-0/testchainca.cert with env {}
W20260806 06:26:43.978730 28919 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 06:26:43.978924 28919 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 06:26:43.978952 28919 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 06:26:43.980069 28919 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 06:26:43.980118 28919 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 06:26:43.980134 28919 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 06:26:43.980149 28919 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 06:26:43.980165 28919 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260806 06:26:43.980204 28919 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260806 06:26:43.980221 28919 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:41543/jwks.json
I20260806 06:26:43.981696 28919 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.148:45907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785997581049705-21554-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785997581049705-21554-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.12.190:41293
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.12.190:41293
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41543/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785997581049705-21554-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.12.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785997581049705-21554-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28919
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785997581049705-21554-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 05:11:02 UTC on ccycloud.quasar-hvzbbz.root.comops.site
build id 4569617
I20260806 06:26:43.981935 28919 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 06:26:43.982121 28919 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 06:26:43.986799 28926 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:26:43.986832 28919 server_base.cc:1034] running on GCE node
W20260806 06:26:43.986900 28925 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:26:43.986901 28929 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:26:43.987211 28919 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 06:26:43.987478 28919 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 06:26:43.988631 28919 hybrid_clock.cc:648] HybridClock initialized: now 1785997603988619 us; error 27 us; skew 500 ppm
I20260806 06:26:43.992417 28919 webserver.cc:460] Webserver started at http://127.21.12.190:46467/ using document root <none> and password file <none>
I20260806 06:26:43.992623 28919 fs_manager.cc:359] Metadata directory not provided
I20260806 06:26:43.992667 28919 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 06:26:43.992758 28919 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 06:26:43.993726 28919 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785997581049705-21554-0/minicluster-data/master-0/data/instance:
uuid: "0fb997f1c2de44b39f2e75fa33bb41ca"
format_stamp: "Formatted at 2026-08-06 06:26:43 on dist-test-slave-4tdj"
server_key: "037412105eaf2cb72be99300a182cfbe"
server_key_iv: "d3857def6b334399d1c2dbf30d4fcc1a"
server_key_version: "encryptionkey@0"
I20260806 06:26:43.994017 28919 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785997581049705-21554-0/minicluster-data/master-0/wal/instance:
uuid: "0fb997f1c2de44b39f2e75fa33bb41ca"
format_stamp: "Formatted at 2026-08-06 06:26:43 on dist-test-slave-4tdj"
server_key: "037412105eaf2cb72be99300a182cfbe"
server_key_iv: "d3857def6b334399d1c2dbf30d4fcc1a"
server_key_version: "encryptionkey@0"
I20260806 06:26:43.995225 28919 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260806 06:26:43.995839 28936 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 06:26:43.995985 28919 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260806 06:26:43.996062 28919 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785997581049705-21554-0/minicluster-data/master-0/data,/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785997581049705-21554-0/minicluster-data/master-0/wal
uuid: "0fb997f1c2de44b39f2e75fa33bb41ca"
format_stamp: "Formatted at 2026-08-06 06:26:43 on dist-test-slave-4tdj"
server_key: "037412105eaf2cb72be99300a182cfbe"
server_key_iv: "d3857def6b334399d1c2dbf30d4fcc1a"
server_key_version: "encryptionkey@0"
I20260806 06:26:43.996136 28919 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785997581049705-21554-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785997581049705-21554-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785997581049705-21554-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 06:26:44.010033 28919 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 06:26:44.010363 28919 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 06:26:44.010489 28919 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 06:26:44.014072 28919 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.190:41293
I20260806 06:26:44.014128 28999 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.190:41293 every 8 connection(s)
I20260806 06:26:44.014446 28919 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785997581049705-21554-0/minicluster-data/master-0/data/info.pb
I20260806 06:26:44.015075 29002 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 06:26:44.017874 29002 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0fb997f1c2de44b39f2e75fa33bb41ca: Bootstrap starting.
I20260806 06:26:44.018443 29002 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0fb997f1c2de44b39f2e75fa33bb41ca: Neither blocks nor log segments found. Creating new log.
I20260806 06:26:44.018700 29002 log.cc:824] T 00000000000000000000000000000000 P 0fb997f1c2de44b39f2e75fa33bb41ca: Log is configured to *not* fsync() on all Append() calls
I20260806 06:26:44.019325 29002 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0fb997f1c2de44b39f2e75fa33bb41ca: No bootstrap required, opened a new log
I20260806 06:26:44.020888 29002 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0fb997f1c2de44b39f2e75fa33bb41ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fb997f1c2de44b39f2e75fa33bb41ca" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 41293 } }
I20260806 06:26:44.021023 29002 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0fb997f1c2de44b39f2e75fa33bb41ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 06:26:44.021044 29002 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0fb997f1c2de44b39f2e75fa33bb41ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fb997f1c2de44b39f2e75fa33bb41ca, State: Initialized, Role: FOLLOWER
I20260806 06:26:44.021147 29002 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0fb997f1c2de44b39f2e75fa33bb41ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fb997f1c2de44b39f2e75fa33bb41ca" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 41293 } }
I20260806 06:26:44.021195 29002 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0fb997f1c2de44b39f2e75fa33bb41ca [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 06:26:44.021217 29002 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0fb997f1c2de44b39f2e75fa33bb41ca [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 06:26:44.021247 29002 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0fb997f1c2de44b39f2e75fa33bb41ca [term 0 FOLLOWER]: Advancing to term 1
I20260806 06:26:44.021728 29002 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0fb997f1c2de44b39f2e75fa33bb41ca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fb997f1c2de44b39f2e75fa33bb41ca" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 41293 } }
I20260806 06:26:44.021812 29002 leader_election.cc:302] T 00000000000000000000000000000000 P 0fb997f1c2de44b39f2e75fa33bb41ca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0fb997f1c2de44b39f2e75fa33bb41ca; no voters:
I20260806 06:26:44.021935 29002 leader_election.cc:288] T 00000000000000000000000000000000 P 0fb997f1c2de44b39f2e75fa33bb41ca [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 06:26:44.021979 29007 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0fb997f1c2de44b39f2e75fa33bb41ca [term 1 FOLLOWER]: Leader election won for term 1
I20260806 06:26:44.022110 29007 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0fb997f1c2de44b39f2e75fa33bb41ca [term 1 LEADER]: Becoming Leader. State: Replica: 0fb997f1c2de44b39f2e75fa33bb41ca, State: Running, Role: LEADER
I20260806 06:26:44.022162 29002 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0fb997f1c2de44b39f2e75fa33bb41ca [sys.catalog]: configured and running, proceeding with master startup.
I20260806 06:26:44.022207 29007 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0fb997f1c2de44b39f2e75fa33bb41ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fb997f1c2de44b39f2e75fa33bb41ca" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 41293 } }
I20260806 06:26:44.022586 29008 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0fb997f1c2de44b39f2e75fa33bb41ca [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0fb997f1c2de44b39f2e75fa33bb41ca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fb997f1c2de44b39f2e75fa33bb41ca" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 41293 } } }
I20260806 06:26:44.022724 29008 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0fb997f1c2de44b39f2e75fa33bb41ca [sys.catalog]: This master's current role is: LEADER
I20260806 06:26:44.022763 29009 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0fb997f1c2de44b39f2e75fa33bb41ca [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0fb997f1c2de44b39f2e75fa33bb41ca. Latest consensus state: current_term: 1 leader_uuid: "0fb997f1c2de44b39f2e75fa33bb41ca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fb997f1c2de44b39f2e75fa33bb41ca" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 41293 } } }
I20260806 06:26:44.022843 29009 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0fb997f1c2de44b39f2e75fa33bb41ca [sys.catalog]: This master's current role is: LEADER
I20260806 06:26:44.023088 29018 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 06:26:44.023587 29018 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 06:26:44.024091 21554 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHhLhT/build/release/bin/kudu as pid 28919
I20260806 06:26:44.024197 21554 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785997581049705-21554-0/minicluster-data/master-0/wal/instance
I20260806 06:26:44.024377 21554 external_mini_cluster.cc:1428] Setting key 295e383a7485069d01c3b92a8ba8e594
I20260806 06:26:44.026739 29018 catalog_manager.cc:1347] Generated new cluster ID: bdc50f3a770241e89b3100f5d24d0d5e
I20260806 06:26:44.026798 29018 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 06:26:44.030699 29033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:44.024769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34642 (local address 127.21.12.190:41293)
0806 06:26:44.026819 (+ 2050us) server_negotiation.cc:206] Beginning negotiation
0806 06:26:44.026829 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 06:26:44.026864 (+ 35us) server_negotiation.cc:407] Connection header received
0806 06:26:44.026957 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:26:44.026963 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:26:44.027021 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:26:44.027139 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:26:44.027340 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:44.028221 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:44.028961 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:44.029358 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:44.029464 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:44.030084 (+ 620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:26:44.030088 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:26:44.030088 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 06:26:44.030249 (+ 161us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:26:44.030280 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 06:26:44.030309 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:26:44.030369 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0806 06:26:44.030440 (+ 71us) server_negotiation.cc:299] Negotiation successful
0806 06:26:44.030518 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1976,"thread_start_us":59,"threads_started":1}
I20260806 06:26:44.031630 21554 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 06:26:44.046398 29018 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 06:26:44.046868 29018 catalog_manager.cc:1504] Loading token signing keys...
I20260806 06:26:44.051143 29018 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0fb997f1c2de44b39f2e75fa33bb41ca: Generated new TSK 0
I20260806 06:26:44.051303 29018 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260806 06:26:44.072659 29041 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260806 06:26:44.072733 29041 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260806 06:26:44.073047 29033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:44.059494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34648 (local address 127.21.12.190:41293)
0806 06:26:44.059541 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0806 06:26:44.059543 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:26:44.059681 (+ 138us) server_negotiation.cc:407] Connection header received
0806 06:26:44.059739 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:26:44.059740 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:26:44.059752 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:26:44.059813 (+ 61us) server_negotiation.cc:226] Negotiated authn=JWT
0806 06:26:44.059931 (+ 118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:44.060426 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:44.061165 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:44.061376 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:44.061428 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:44.061586 (+ 158us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0806 06:26:44.072843 (+ 11257us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41543/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0806 06:26:44.072976 (+ 133us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:34648: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41543/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":26}
W20260806 06:26:44.073113 29040 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 06:26:44.059437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:41293 (local address 127.0.0.1:34648)
0806 06:26:44.059615 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0806 06:26:44.059624 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0806 06:26:44.059682 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:26:44.059817 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:26:44.059818 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:26:44.059819 (+ 1us) client_negotiation.cc:189] Negotiated authn=JWT
0806 06:26:44.059873 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:44.059875 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:44.060499 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:44.060500 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:44.061085 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:44.061087 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:44.061442 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:44.061444 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:44.061499 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:44.061503 (+ 4us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0806 06:26:44.072930 (+ 11427us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41543/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0806 06:26:44.073029 (+ 99us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.21.12.190:41293: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41543/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":113,"threads_started":1}
W20260806 06:26:44.073115 29033 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:34648: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41543/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
I20260806 06:26:44.073266 21554 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHhLhT/build/release/bin/kudu with pid 28919
2026-08-06T06:26:44Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (233 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260806 06:26:44.124949 21554 mini_oidc.cc:152] Starting JWKS server
I20260806 06:26:44.125643 21554 webserver.cc:460] Webserver started at https://127.0.0.1:33801/ using document root <none> and password file <none>
I20260806 06:26:44.125689 21554 mini_oidc.cc:189] Starting OIDC Discovery server
I20260806 06:26:44.125824 21554 webserver.cc:460] Webserver started at http://127.0.0.1:46691/ using document root <none> and password file <none>
2026-08-06T06:26:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T06:26:44Z Disabled control of system clock
I20260806 06:26:44.133900 21554 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHhLhT/build/release/bin/kudu
/tmp/dist-test-taskFHhLhT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785997581049705-21554-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785997581049705-21554-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785997581049705-21554-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785997581049705-21554-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.12.190:37337
--webserver_interface=127.21.12.190
--webserver_port=0
--builtin_ntp_servers=127.21.12.148:37379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.12.190:37337
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33801/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260806 06:26:44.212518 29052 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 06:26:44.212741 29052 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 06:26:44.212772 29052 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 06:26:44.213913 29052 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 06:26:44.213953 29052 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 06:26:44.213968 29052 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 06:26:44.213985 29052 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 06:26:44.213999 29052 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260806 06:26:44.214022 29052 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260806 06:26:44.214041 29052 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:33801/jwks.json
I20260806 06:26:44.215409 29052 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.148:37379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785997581049705-21554-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785997581049705-21554-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.12.190:37337
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.12.190:37337
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33801/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785997581049705-21554-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.12.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29052
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785997581049705-21554-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 05:11:02 UTC on ccycloud.quasar-hvzbbz.root.comops.site
build id 4569617
I20260806 06:26:44.215660 29052 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 06:26:44.215919 29052 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 06:26:44.220640 29059 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:26:44.220650 29061 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:26:44.220650 29063 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:26:44.220685 29052 server_base.cc:1034] running on GCE node
I20260806 06:26:44.221081 29052 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 06:26:44.221355 29052 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 06:26:44.222488 29052 hybrid_clock.cc:648] HybridClock initialized: now 1785997604222472 us; error 34 us; skew 500 ppm
I20260806 06:26:44.223619 29052 webserver.cc:460] Webserver started at http://127.21.12.190:41941/ using document root <none> and password file <none>
I20260806 06:26:44.223815 29052 fs_manager.cc:359] Metadata directory not provided
I20260806 06:26:44.223858 29052 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 06:26:44.223980 29052 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 06:26:44.224994 29052 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785997581049705-21554-0/minicluster-data/master-0/data/instance:
uuid: "d0a5fd0d029040d39933aa2cbca493aa"
format_stamp: "Formatted at 2026-08-06 06:26:44 on dist-test-slave-4tdj"
server_key: "d3480d86f76f0535b52e5bd1ecb69b60"
server_key_iv: "15bf1b9e156c4c7ac6a5ca139cd37b9c"
server_key_version: "encryptionkey@0"
I20260806 06:26:44.225345 29052 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785997581049705-21554-0/minicluster-data/master-0/wal/instance:
uuid: "d0a5fd0d029040d39933aa2cbca493aa"
format_stamp: "Formatted at 2026-08-06 06:26:44 on dist-test-slave-4tdj"
server_key: "d3480d86f76f0535b52e5bd1ecb69b60"
server_key_iv: "15bf1b9e156c4c7ac6a5ca139cd37b9c"
server_key_version: "encryptionkey@0"
I20260806 06:26:44.226567 29052 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260806 06:26:44.227213 29073 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 06:26:44.227349 29052 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260806 06:26:44.227428 29052 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785997581049705-21554-0/minicluster-data/master-0/data,/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785997581049705-21554-0/minicluster-data/master-0/wal
uuid: "d0a5fd0d029040d39933aa2cbca493aa"
format_stamp: "Formatted at 2026-08-06 06:26:44 on dist-test-slave-4tdj"
server_key: "d3480d86f76f0535b52e5bd1ecb69b60"
server_key_iv: "15bf1b9e156c4c7ac6a5ca139cd37b9c"
server_key_version: "encryptionkey@0"
I20260806 06:26:44.227491 29052 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785997581049705-21554-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785997581049705-21554-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785997581049705-21554-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 06:26:44.242882 29052 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 06:26:44.243178 29052 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 06:26:44.243304 29052 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 06:26:44.246965 29052 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.190:37337
I20260806 06:26:44.246999 29154 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.190:37337 every 8 connection(s)
I20260806 06:26:44.247310 29052 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785997581049705-21554-0/minicluster-data/master-0/data/info.pb
I20260806 06:26:44.247918 21554 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHhLhT/build/release/bin/kudu as pid 29052
I20260806 06:26:44.248032 21554 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785997581049705-21554-0/minicluster-data/master-0/wal/instance
I20260806 06:26:44.248098 29155 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 06:26:44.248245 21554 external_mini_cluster.cc:1428] Setting key f96227acdd452f1f9f0471fbc69cb14a
I20260806 06:26:44.251708 29155 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0a5fd0d029040d39933aa2cbca493aa: Bootstrap starting.
I20260806 06:26:44.252373 29155 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d0a5fd0d029040d39933aa2cbca493aa: Neither blocks nor log segments found. Creating new log.
I20260806 06:26:44.252635 29155 log.cc:824] T 00000000000000000000000000000000 P d0a5fd0d029040d39933aa2cbca493aa: Log is configured to *not* fsync() on all Append() calls
I20260806 06:26:44.253331 29155 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0a5fd0d029040d39933aa2cbca493aa: No bootstrap required, opened a new log
I20260806 06:26:44.253857 29159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:44.248759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45154 (local address 127.21.12.190:37337)
0806 06:26:44.248941 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0806 06:26:44.248949 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 06:26:44.248979 (+ 30us) server_negotiation.cc:407] Connection header received
0806 06:26:44.249727 (+ 748us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:26:44.249736 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:26:44.249800 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:26:44.249912 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:26:44.250135 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:44.251045 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:44.252055 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:44.252456 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:44.252574 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:44.253216 (+ 642us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:26:44.253220 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:26:44.253221 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 06:26:44.253381 (+ 160us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:26:44.253416 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 06:26:44.253430 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:26:44.253481 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0806 06:26:44.253578 (+ 97us) server_negotiation.cc:299] Negotiation successful
0806 06:26:44.253694 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":67,"threads_started":1}
I20260806 06:26:44.254809 21554 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 06:26:44.255437 29155 raft_consensus.cc:348] T 00000000000000000000000000000000 P d0a5fd0d029040d39933aa2cbca493aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0a5fd0d029040d39933aa2cbca493aa" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 37337 } }
I20260806 06:26:44.255613 29155 raft_consensus.cc:374] T 00000000000000000000000000000000 P d0a5fd0d029040d39933aa2cbca493aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 06:26:44.255661 29155 raft_consensus.cc:729] T 00000000000000000000000000000000 P d0a5fd0d029040d39933aa2cbca493aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0a5fd0d029040d39933aa2cbca493aa, State: Initialized, Role: FOLLOWER
I20260806 06:26:44.255771 29155 consensus_queue.cc:260] T 00000000000000000000000000000000 P d0a5fd0d029040d39933aa2cbca493aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0a5fd0d029040d39933aa2cbca493aa" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 37337 } }
I20260806 06:26:44.255860 29155 raft_consensus.cc:388] T 00000000000000000000000000000000 P d0a5fd0d029040d39933aa2cbca493aa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 06:26:44.255895 29155 raft_consensus.cc:482] T 00000000000000000000000000000000 P d0a5fd0d029040d39933aa2cbca493aa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 06:26:44.255928 29155 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d0a5fd0d029040d39933aa2cbca493aa [term 0 FOLLOWER]: Advancing to term 1
I20260806 06:26:44.256428 29155 raft_consensus.cc:504] T 00000000000000000000000000000000 P d0a5fd0d029040d39933aa2cbca493aa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0a5fd0d029040d39933aa2cbca493aa" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 37337 } }
I20260806 06:26:44.256516 29155 leader_election.cc:302] T 00000000000000000000000000000000 P d0a5fd0d029040d39933aa2cbca493aa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d0a5fd0d029040d39933aa2cbca493aa; no voters:
I20260806 06:26:44.256649 29155 leader_election.cc:288] T 00000000000000000000000000000000 P d0a5fd0d029040d39933aa2cbca493aa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 06:26:44.256707 29162 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d0a5fd0d029040d39933aa2cbca493aa [term 1 FOLLOWER]: Leader election won for term 1
I20260806 06:26:44.256820 29162 raft_consensus.cc:686] T 00000000000000000000000000000000 P d0a5fd0d029040d39933aa2cbca493aa [term 1 LEADER]: Becoming Leader. State: Replica: d0a5fd0d029040d39933aa2cbca493aa, State: Running, Role: LEADER
I20260806 06:26:44.256907 29155 sys_catalog.cc:564] T 00000000000000000000000000000000 P d0a5fd0d029040d39933aa2cbca493aa [sys.catalog]: configured and running, proceeding with master startup.
I20260806 06:26:44.256899 29162 consensus_queue.cc:237] T 00000000000000000000000000000000 P d0a5fd0d029040d39933aa2cbca493aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0a5fd0d029040d39933aa2cbca493aa" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 37337 } }
I20260806 06:26:44.257294 29163 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0a5fd0d029040d39933aa2cbca493aa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d0a5fd0d029040d39933aa2cbca493aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0a5fd0d029040d39933aa2cbca493aa" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 37337 } } }
I20260806 06:26:44.257305 29164 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0a5fd0d029040d39933aa2cbca493aa [sys.catalog]: SysCatalogTable state changed. Reason: New leader d0a5fd0d029040d39933aa2cbca493aa. Latest consensus state: current_term: 1 leader_uuid: "d0a5fd0d029040d39933aa2cbca493aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0a5fd0d029040d39933aa2cbca493aa" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 37337 } } }
I20260806 06:26:44.257465 29163 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0a5fd0d029040d39933aa2cbca493aa [sys.catalog]: This master's current role is: LEADER
I20260806 06:26:44.257481 29164 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0a5fd0d029040d39933aa2cbca493aa [sys.catalog]: This master's current role is: LEADER
I20260806 06:26:44.257912 29176 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 06:26:44.258416 29176 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 06:26:44.259820 29176 catalog_manager.cc:1347] Generated new cluster ID: 8d5f311dd89f4e2fb6e193d8ab56b003
I20260806 06:26:44.259871 29176 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 06:26:44.274175 29176 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 06:26:44.274691 29176 catalog_manager.cc:1504] Loading token signing keys...
I20260806 06:26:44.278806 29176 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d0a5fd0d029040d39933aa2cbca493aa: Generated new TSK 0
I20260806 06:26:44.278954 29176 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 06:26:44.287127 29159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:44.284113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45156 (local address 127.21.12.190:37337)
0806 06:26:44.284157 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0806 06:26:44.284159 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:26:44.284239 (+ 80us) server_negotiation.cc:407] Connection header received
0806 06:26:44.284305 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:26:44.284306 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:26:44.284318 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:26:44.284377 (+ 59us) server_negotiation.cc:226] Negotiated authn=JWT
0806 06:26:44.284517 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:44.284990 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:44.285793 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:44.285982 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:44.286041 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:44.286192 (+ 151us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0806 06:26:44.286940 (+ 748us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33801/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0806 06:26:44.287067 (+ 127us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:45156: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33801/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":24}
W20260806 06:26:44.287179 29159 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:45156: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33801/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260806 06:26:44.287216 29194 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 06:26:44.284010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:37337 (local address 127.0.0.1:45156)
0806 06:26:44.284163 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0806 06:26:44.284174 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0806 06:26:44.284242 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:26:44.284383 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:26:44.284384 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:26:44.284386 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0806 06:26:44.284455 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:44.284457 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:44.285072 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:44.285073 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:44.285707 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:44.285710 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:44.286043 (+ 333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:44.286044 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:44.286101 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:44.286106 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0806 06:26:44.287029 (+ 923us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33801/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0806 06:26:44.287121 (+ 92us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.21.12.190:37337: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33801/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":85,"threads_started":1}
I20260806 06:26:44.287413 21554 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHhLhT/build/release/bin/kudu with pid 29052
E20260806 06:26:44.309113 29195 webserver.cc:533] Webserver: sslize failed:
E20260806 06:26:44.309194 29195 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-08-06T06:26:44Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (211 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-08-06T06:26:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T06:26:44Z Disabled control of system clock
I20260806 06:26:44.344610 21554 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHhLhT/build/release/bin/kudu
/tmp/dist-test-taskFHhLhT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785997581049705-21554-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785997581049705-21554-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785997581049705-21554-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785997581049705-21554-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.12.190:43115
--webserver_interface=127.21.12.190
--webserver_port=0
--builtin_ntp_servers=127.21.12.148:44235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.12.190:43115
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785997581049705-21554-0/testchainca.cert with env {}
W20260806 06:26:44.424309 29204 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 06:26:44.424503 29204 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 06:26:44.424533 29204 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 06:26:44.425681 29204 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 06:26:44.425720 29204 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 06:26:44.425737 29204 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 06:26:44.425753 29204 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 06:26:44.425768 29204 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260806 06:26:44.425791 29204 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260806 06:26:44.425807 29204 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260806 06:26:44.427667 29204 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.148:44235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785997581049705-21554-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785997581049705-21554-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.12.190:43115
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.12.190:43115
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785997581049705-21554-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.12.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785997581049705-21554-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29204
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785997581049705-21554-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 05:11:02 UTC on ccycloud.quasar-hvzbbz.root.comops.site
build id 4569617
I20260806 06:26:44.427920 29204 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 06:26:44.428159 29204 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 06:26:44.432781 29213 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:26:44.432838 29210 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:26:44.432889 29209 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:26:44.432904 29204 server_base.cc:1034] running on GCE node
I20260806 06:26:44.433233 29204 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 06:26:44.433502 29204 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 06:26:44.434675 29204 hybrid_clock.cc:648] HybridClock initialized: now 1785997604434661 us; error 25 us; skew 500 ppm
I20260806 06:26:44.435878 29204 webserver.cc:460] Webserver started at http://127.21.12.190:39455/ using document root <none> and password file <none>
I20260806 06:26:44.436094 29204 fs_manager.cc:359] Metadata directory not provided
I20260806 06:26:44.436156 29204 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 06:26:44.436281 29204 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 06:26:44.437288 29204 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785997581049705-21554-0/minicluster-data/master-0/data/instance:
uuid: "446a227bef6347e481116deaa4a74678"
format_stamp: "Formatted at 2026-08-06 06:26:44 on dist-test-slave-4tdj"
server_key: "9cf83ced43d5291525b109e79f088f29"
server_key_iv: "c5a0ccfa907dc31b5969d445b45ad34e"
server_key_version: "encryptionkey@0"
I20260806 06:26:44.437629 29204 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785997581049705-21554-0/minicluster-data/master-0/wal/instance:
uuid: "446a227bef6347e481116deaa4a74678"
format_stamp: "Formatted at 2026-08-06 06:26:44 on dist-test-slave-4tdj"
server_key: "9cf83ced43d5291525b109e79f088f29"
server_key_iv: "c5a0ccfa907dc31b5969d445b45ad34e"
server_key_version: "encryptionkey@0"
I20260806 06:26:44.438901 29204 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260806 06:26:44.439534 29222 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 06:26:44.439685 29204 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260806 06:26:44.439770 29204 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785997581049705-21554-0/minicluster-data/master-0/data,/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785997581049705-21554-0/minicluster-data/master-0/wal
uuid: "446a227bef6347e481116deaa4a74678"
format_stamp: "Formatted at 2026-08-06 06:26:44 on dist-test-slave-4tdj"
server_key: "9cf83ced43d5291525b109e79f088f29"
server_key_iv: "c5a0ccfa907dc31b5969d445b45ad34e"
server_key_version: "encryptionkey@0"
I20260806 06:26:44.439831 29204 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785997581049705-21554-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785997581049705-21554-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785997581049705-21554-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 06:26:44.447108 29204 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 06:26:44.447347 29204 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 06:26:44.447460 29204 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 06:26:44.451179 29204 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.190:43115
I20260806 06:26:44.451211 29297 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.190:43115 every 8 connection(s)
I20260806 06:26:44.451495 29204 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785997581049705-21554-0/minicluster-data/master-0/data/info.pb
I20260806 06:26:44.452373 29298 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 06:26:44.454558 29298 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 446a227bef6347e481116deaa4a74678: Bootstrap starting.
I20260806 06:26:44.455060 29298 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 446a227bef6347e481116deaa4a74678: Neither blocks nor log segments found. Creating new log.
I20260806 06:26:44.455287 29298 log.cc:824] T 00000000000000000000000000000000 P 446a227bef6347e481116deaa4a74678: Log is configured to *not* fsync() on all Append() calls
I20260806 06:26:44.455885 29298 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 446a227bef6347e481116deaa4a74678: No bootstrap required, opened a new log
I20260806 06:26:44.457406 29298 raft_consensus.cc:348] T 00000000000000000000000000000000 P 446a227bef6347e481116deaa4a74678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "446a227bef6347e481116deaa4a74678" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 43115 } }
I20260806 06:26:44.457551 29298 raft_consensus.cc:374] T 00000000000000000000000000000000 P 446a227bef6347e481116deaa4a74678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 06:26:44.457571 29298 raft_consensus.cc:729] T 00000000000000000000000000000000 P 446a227bef6347e481116deaa4a74678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 446a227bef6347e481116deaa4a74678, State: Initialized, Role: FOLLOWER
I20260806 06:26:44.457675 29298 consensus_queue.cc:260] T 00000000000000000000000000000000 P 446a227bef6347e481116deaa4a74678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "446a227bef6347e481116deaa4a74678" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 43115 } }
I20260806 06:26:44.457751 29298 raft_consensus.cc:388] T 00000000000000000000000000000000 P 446a227bef6347e481116deaa4a74678 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 06:26:44.457784 29298 raft_consensus.cc:482] T 00000000000000000000000000000000 P 446a227bef6347e481116deaa4a74678 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 06:26:44.457816 29298 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 446a227bef6347e481116deaa4a74678 [term 0 FOLLOWER]: Advancing to term 1
I20260806 06:26:44.458338 29298 raft_consensus.cc:504] T 00000000000000000000000000000000 P 446a227bef6347e481116deaa4a74678 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "446a227bef6347e481116deaa4a74678" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 43115 } }
I20260806 06:26:44.458448 29298 leader_election.cc:302] T 00000000000000000000000000000000 P 446a227bef6347e481116deaa4a74678 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 446a227bef6347e481116deaa4a74678; no voters:
I20260806 06:26:44.458616 29298 leader_election.cc:288] T 00000000000000000000000000000000 P 446a227bef6347e481116deaa4a74678 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 06:26:44.458628 21554 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHhLhT/build/release/bin/kudu as pid 29204
I20260806 06:26:44.458673 29301 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 446a227bef6347e481116deaa4a74678 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 06:26:44.458724 21554 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785997581049705-21554-0/minicluster-data/master-0/wal/instance
I20260806 06:26:44.458817 29301 raft_consensus.cc:686] T 00000000000000000000000000000000 P 446a227bef6347e481116deaa4a74678 [term 1 LEADER]: Becoming Leader. State: Replica: 446a227bef6347e481116deaa4a74678, State: Running, Role: LEADER
I20260806 06:26:44.458864 29298 sys_catalog.cc:564] T 00000000000000000000000000000000 P 446a227bef6347e481116deaa4a74678 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 06:26:44.458920 21554 external_mini_cluster.cc:1428] Setting key b6d216c769ff033f0f9b23cdb522a503
I20260806 06:26:44.458942 29301 consensus_queue.cc:237] T 00000000000000000000000000000000 P 446a227bef6347e481116deaa4a74678 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "446a227bef6347e481116deaa4a74678" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 43115 } }
I20260806 06:26:44.459290 29303 sys_catalog.cc:455] T 00000000000000000000000000000000 P 446a227bef6347e481116deaa4a74678 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "446a227bef6347e481116deaa4a74678" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "446a227bef6347e481116deaa4a74678" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 43115 } } }
I20260806 06:26:44.459304 29305 sys_catalog.cc:455] T 00000000000000000000000000000000 P 446a227bef6347e481116deaa4a74678 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 446a227bef6347e481116deaa4a74678. Latest consensus state: current_term: 1 leader_uuid: "446a227bef6347e481116deaa4a74678" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "446a227bef6347e481116deaa4a74678" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 43115 } } }
I20260806 06:26:44.459384 29303 sys_catalog.cc:458] T 00000000000000000000000000000000 P 446a227bef6347e481116deaa4a74678 [sys.catalog]: This master's current role is: LEADER
I20260806 06:26:44.459388 29305 sys_catalog.cc:458] T 00000000000000000000000000000000 P 446a227bef6347e481116deaa4a74678 [sys.catalog]: This master's current role is: LEADER
I20260806 06:26:44.459724 29308 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 06:26:44.460210 29308 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 06:26:44.461961 29308 catalog_manager.cc:1347] Generated new cluster ID: 615bf94508784925a9f7299cbd12befd
I20260806 06:26:44.462028 29308 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 06:26:44.464824 29314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:44.459697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48220 (local address 127.21.12.190:43115)
0806 06:26:44.459941 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0806 06:26:44.459950 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 06:26:44.459993 (+ 43us) server_negotiation.cc:407] Connection header received
0806 06:26:44.460107 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:26:44.460115 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:26:44.460189 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:26:44.460591 (+ 402us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:26:44.460691 (+ 100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:44.461702 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:44.462775 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:44.463392 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:44.463521 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:44.464123 (+ 602us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:26:44.464128 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:26:44.464128 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 06:26:44.464341 (+ 213us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:26:44.464383 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 06:26:44.464399 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:26:44.464470 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0806 06:26:44.464534 (+ 64us) server_negotiation.cc:299] Negotiation successful
0806 06:26:44.464619 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":81,"threads_started":1}
I20260806 06:26:44.465983 21554 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 06:26:44.468808 29308 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 06:26:44.469457 29308 catalog_manager.cc:1504] Loading token signing keys...
I20260806 06:26:44.478601 29308 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 446a227bef6347e481116deaa4a74678: Generated new TSK 0
I20260806 06:26:44.478756 29308 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 06:26:44.592644 29314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:44.489581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48236 (local address 127.21.12.190:43115)
0806 06:26:44.489631 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0806 06:26:44.489633 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:26:44.489840 (+ 207us) server_negotiation.cc:407] Connection header received
0806 06:26:44.489890 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:26:44.489890 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:26:44.489903 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:26:44.489977 (+ 74us) server_negotiation.cc:226] Negotiated authn=JWT
0806 06:26:44.490066 (+ 89us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:44.490869 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:44.491586 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:44.491880 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:44.491939 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:44.492080 (+ 141us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0806 06:26:44.592423 (+100343us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0806 06:26:44.592585 (+ 162us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:48236: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":29}
W20260806 06:26:44.592707 29314 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:48236: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260806 06:26:44.592806 29340 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 06:26:44.489573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:43115 (local address 127.0.0.1:48236)
0806 06:26:44.489769 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0806 06:26:44.489780 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0806 06:26:44.489840 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:26:44.489981 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:26:44.489981 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:26:44.489983 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0806 06:26:44.490026 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:44.490028 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:44.490938 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:44.490938 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:44.491512 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:44.491514 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:44.491948 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:44.491948 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:44.491999 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:44.492007 (+ 8us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0806 06:26:44.592552 (+100545us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0806 06:26:44.592689 (+ 137us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.21.12.190:43115: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":124,"threads_started":1}
I20260806 06:26:44.593230 21554 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHhLhT/build/release/bin/kudu with pid 29204
2026-08-06T06:26:44Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (276 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (87 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (86 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785997581049705-21554-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 06:26:44 dist-test-slave-4tdj krb5kdc[29356](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 06:26:44 dist-test-slave-4tdj krb5kdc[29356](info): set up 2 sockets
Aug 06 06:26:44 dist-test-slave-4tdj krb5kdc[29356](info): commencing operation
krb5kdc: starting...
W20260806 06:26:45.881036 21554 mini_kdc.cc:121] Time spent starting KDC: real 1.094s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785997581049705-21554-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785997581049705-21554-0/krb5kdc/test-user.keytab.
Aug 06 06:26:45 dist-test-slave-4tdj krb5kdc[29356](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997605, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T06:26:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T06:26:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.12.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.12.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.12.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.190.keytab.
Entry for principal kudu/127.21.12.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.190.keytab.
I20260806 06:26:45.995136 21554 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHhLhT/build/release/bin/kudu
/tmp/dist-test-taskFHhLhT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785997581049705-21554-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785997581049705-21554-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785997581049705-21554-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785997581049705-21554-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.12.190:35581
--webserver_interface=127.21.12.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.190.keytab
--principal=kudu/127.21.12.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.148:44239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.12.190:35581
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785997581049705-21554-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785997581049705-21554-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785997581049705-21554-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 06:26:46.074563 29374 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 06:26:46.074764 29374 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 06:26:46.074793 29374 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 06:26:46.075927 29374 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 06:26:46.075968 29374 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 06:26:46.075985 29374 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 06:26:46.076000 29374 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 06:26:46.076015 29374 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 06:26:46.077431 29374 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.148:44239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785997581049705-21554-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785997581049705-21554-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.12.190:35581
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.190.keytab
--principal=kudu/127.21.12.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.12.190:35581
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785997581049705-21554-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.12.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29374
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785997581049705-21554-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 05:11:02 UTC on ccycloud.quasar-hvzbbz.root.comops.site
build id 4569617
I20260806 06:26:46.077674 29374 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 06:26:46.077868 29374 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260806 06:26:46.082525 29374 server_base.cc:1034] running on GCE node
W20260806 06:26:46.082551 29383 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:26:46.082540 29381 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:26:46.082665 29380 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:26:46.082898 29374 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 06:26:46.083168 29374 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 06:26:46.084304 29374 hybrid_clock.cc:648] HybridClock initialized: now 1785997606084292 us; error 31 us; skew 500 ppm
Aug 06 06:26:46 dist-test-slave-4tdj krb5kdc[29356](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997606, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 06:26:46.086683 29374 init.cc:377] Logged in from keytab as kudu/127.21.12.190@KRBTEST.COM (short username kudu)
I20260806 06:26:46.087100 29374 webserver.cc:460] Webserver started at http://127.21.12.190:33813/ using document root <none> and password file <none>
I20260806 06:26:46.087292 29374 fs_manager.cc:359] Metadata directory not provided
I20260806 06:26:46.087332 29374 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 06:26:46.087422 29374 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 06:26:46.088369 29374 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785997581049705-21554-0/minicluster-data/master-0/data/instance:
uuid: "1b8e27526c7b430397086d080eee7d15"
format_stamp: "Formatted at 2026-08-06 06:26:46 on dist-test-slave-4tdj"
server_key: "9caab3fbb3aab1e03a450249d9f44545"
server_key_iv: "677f8f4a9e6afb4bfedda11792653f5c"
server_key_version: "encryptionkey@0"
I20260806 06:26:46.088656 29374 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785997581049705-21554-0/minicluster-data/master-0/wal/instance:
uuid: "1b8e27526c7b430397086d080eee7d15"
format_stamp: "Formatted at 2026-08-06 06:26:46 on dist-test-slave-4tdj"
server_key: "9caab3fbb3aab1e03a450249d9f44545"
server_key_iv: "677f8f4a9e6afb4bfedda11792653f5c"
server_key_version: "encryptionkey@0"
I20260806 06:26:46.089798 29374 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260806 06:26:46.090483 29394 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 06:26:46.090643 29374 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260806 06:26:46.090718 29374 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785997581049705-21554-0/minicluster-data/master-0/data,/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785997581049705-21554-0/minicluster-data/master-0/wal
uuid: "1b8e27526c7b430397086d080eee7d15"
format_stamp: "Formatted at 2026-08-06 06:26:46 on dist-test-slave-4tdj"
server_key: "9caab3fbb3aab1e03a450249d9f44545"
server_key_iv: "677f8f4a9e6afb4bfedda11792653f5c"
server_key_version: "encryptionkey@0"
I20260806 06:26:46.090777 29374 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785997581049705-21554-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785997581049705-21554-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785997581049705-21554-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 06:26:46.124927 29374 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 06:26:46.127115 29374 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 06:26:46.127254 29374 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 06:26:46.131047 29374 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.190:35581
I20260806 06:26:46.131403 29374 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785997581049705-21554-0/minicluster-data/master-0/data/info.pb
I20260806 06:26:46.131410 29467 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.190:35581 every 8 connection(s)
I20260806 06:26:46.132064 29468 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 06:26:46.134256 29468 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b8e27526c7b430397086d080eee7d15: Bootstrap starting.
I20260806 06:26:46.134819 29468 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1b8e27526c7b430397086d080eee7d15: Neither blocks nor log segments found. Creating new log.
I20260806 06:26:46.135051 29468 log.cc:824] T 00000000000000000000000000000000 P 1b8e27526c7b430397086d080eee7d15: Log is configured to *not* fsync() on all Append() calls
I20260806 06:26:46.135653 29468 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b8e27526c7b430397086d080eee7d15: No bootstrap required, opened a new log
I20260806 06:26:46.137171 29468 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1b8e27526c7b430397086d080eee7d15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b8e27526c7b430397086d080eee7d15" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 35581 } }
I20260806 06:26:46.137307 29468 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1b8e27526c7b430397086d080eee7d15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 06:26:46.137328 29468 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1b8e27526c7b430397086d080eee7d15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b8e27526c7b430397086d080eee7d15, State: Initialized, Role: FOLLOWER
I20260806 06:26:46.137437 29468 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1b8e27526c7b430397086d080eee7d15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b8e27526c7b430397086d080eee7d15" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 35581 } }
I20260806 06:26:46.137511 29468 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1b8e27526c7b430397086d080eee7d15 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 06:26:46.137538 29468 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1b8e27526c7b430397086d080eee7d15 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 06:26:46.137569 29468 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1b8e27526c7b430397086d080eee7d15 [term 0 FOLLOWER]: Advancing to term 1
I20260806 06:26:46.138013 29468 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1b8e27526c7b430397086d080eee7d15 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b8e27526c7b430397086d080eee7d15" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 35581 } }
I20260806 06:26:46.138098 29468 leader_election.cc:302] T 00000000000000000000000000000000 P 1b8e27526c7b430397086d080eee7d15 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1b8e27526c7b430397086d080eee7d15; no voters:
I20260806 06:26:46.138226 29468 leader_election.cc:288] T 00000000000000000000000000000000 P 1b8e27526c7b430397086d080eee7d15 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 06:26:46.138281 29472 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1b8e27526c7b430397086d080eee7d15 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 06:26:46.138474 29468 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1b8e27526c7b430397086d080eee7d15 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 06:26:46.138659 29472 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1b8e27526c7b430397086d080eee7d15 [term 1 LEADER]: Becoming Leader. State: Replica: 1b8e27526c7b430397086d080eee7d15, State: Running, Role: LEADER
I20260806 06:26:46.138756 29472 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1b8e27526c7b430397086d080eee7d15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b8e27526c7b430397086d080eee7d15" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 35581 } }
I20260806 06:26:46.139107 29473 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b8e27526c7b430397086d080eee7d15 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1b8e27526c7b430397086d080eee7d15" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b8e27526c7b430397086d080eee7d15" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 35581 } } }
I20260806 06:26:46.139191 29473 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b8e27526c7b430397086d080eee7d15 [sys.catalog]: This master's current role is: LEADER
I20260806 06:26:46.139353 29474 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b8e27526c7b430397086d080eee7d15 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1b8e27526c7b430397086d080eee7d15. Latest consensus state: current_term: 1 leader_uuid: "1b8e27526c7b430397086d080eee7d15" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b8e27526c7b430397086d080eee7d15" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 35581 } } }
I20260806 06:26:46.139413 29474 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b8e27526c7b430397086d080eee7d15 [sys.catalog]: This master's current role is: LEADER
I20260806 06:26:46.139379 29487 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 06:26:46.139842 21554 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHhLhT/build/release/bin/kudu as pid 29374
I20260806 06:26:46.139937 21554 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785997581049705-21554-0/minicluster-data/master-0/wal/instance
I20260806 06:26:46.140132 21554 external_mini_cluster.cc:1428] Setting key b68099d199809bca106f2863f3de6f6f
I20260806 06:26:46.140656 29487 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 06:26:46.142077 29487 catalog_manager.cc:1347] Generated new cluster ID: 772d1aa9af7749488e80e02fa86831de
I20260806 06:26:46.142120 29487 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 06 06:26:46 dist-test-slave-4tdj krb5kdc[29356](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997605, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.12.190@KRBTEST.COM
I20260806 06:26:46.156998 29499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:46.140562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53840 (local address 127.21.12.190:35581)
0806 06:26:46.140718 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0806 06:26:46.140728 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 06:26:46.146552 (+ 5824us) server_negotiation.cc:407] Connection header received
0806 06:26:46.146713 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:26:46.146720 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:26:46.146775 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:26:46.146915 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:26:46.147667 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:46.149051 (+ 1384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:46.150107 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:46.150769 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:46.150860 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:46.153543 (+ 2683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:26:46.153548 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:26:46.153548 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 06:26:46.153560 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:26:46.155456 (+ 1896us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:26:46.155830 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:26:46.155833 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:26:46.155833 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:26:46.155940 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:26:46.156106 (+ 166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:26:46.156107 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:26:46.156107 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:26:46.156400 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:26:46.156577 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0806 06:26:46.156642 (+ 65us) server_negotiation.cc:299] Negotiation successful
0806 06:26:46.156775 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":76,"threads_started":1}
I20260806 06:26:46.162815 29487 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 06:26:46.163283 29487 catalog_manager.cc:1504] Loading token signing keys...
I20260806 06:26:46.166265 29487 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1b8e27526c7b430397086d080eee7d15: Generated new TSK 0
I20260806 06:26:46.166435 29487 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.12.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.12.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.12.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.129.keytab.
Entry for principal kudu/127.21.12.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.129.keytab.
I20260806 06:26:46.200501 21554 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHhLhT/build/release/bin/kudu
/tmp/dist-test-taskFHhLhT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785997581049705-21554-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785997581049705-21554-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785997581049705-21554-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785997581049705-21554-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.12.129:0
--local_ip_for_outbound_sockets=127.21.12.129
--webserver_interface=127.21.12.129
--webserver_port=0
--tserver_master_addrs=127.21.12.190:35581
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.129.keytab
--principal=kudu/127.21.12.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.148:44239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785997581049705-21554-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785997581049705-21554-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785997581049705-21554-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 06:26:46.302423 29509 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 06:26:46.302605 29509 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 06:26:46.302626 29509 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 06:26:46.303727 29509 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 06:26:46.303761 29509 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 06:26:46.303785 29509 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.12.129
I20260806 06:26:46.305195 29509 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.148:44239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785997581049705-21554-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785997581049705-21554-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.129.keytab
--principal=kudu/127.21.12.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.12.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785997581049705-21554-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.12.129
--webserver_port=0
--tserver_master_addrs=127.21.12.190:35581
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29509
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.12.129
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785997581049705-21554-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 05:11:02 UTC on ccycloud.quasar-hvzbbz.root.comops.site
build id 4569617
I20260806 06:26:46.305398 29509 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 06:26:46.305572 29509 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 06:26:46.310822 29520 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:26:46.310899 29509 server_base.cc:1034] running on GCE node
W20260806 06:26:46.310832 29518 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:26:46.310811 29515 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:26:46.311220 29509 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 06:26:46.311426 29509 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 06:26:46.312584 29509 hybrid_clock.cc:648] HybridClock initialized: now 1785997606312568 us; error 28 us; skew 500 ppm
Aug 06 06:26:46 dist-test-slave-4tdj krb5kdc[29356](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997606, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 06:26:46.315064 29509 init.cc:377] Logged in from keytab as kudu/127.21.12.129@KRBTEST.COM (short username kudu)
I20260806 06:26:46.315462 29509 webserver.cc:460] Webserver started at http://127.21.12.129:39157/ using document root <none> and password file <none>
I20260806 06:26:46.315711 29509 fs_manager.cc:359] Metadata directory not provided
I20260806 06:26:46.315761 29509 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 06:26:46.315887 29509 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 06:26:46.316922 29509 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785997581049705-21554-0/minicluster-data/ts-0/data/instance:
uuid: "3c557ebd102846e19d621703a58a7074"
format_stamp: "Formatted at 2026-08-06 06:26:46 on dist-test-slave-4tdj"
server_key: "a03bfd3cd127a4f3c59f9dc441ca5eaa"
server_key_iv: "f00adc1b1678db1c93e34559c3abe8c8"
server_key_version: "encryptionkey@0"
I20260806 06:26:46.317241 29509 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785997581049705-21554-0/minicluster-data/ts-0/wal/instance:
uuid: "3c557ebd102846e19d621703a58a7074"
format_stamp: "Formatted at 2026-08-06 06:26:46 on dist-test-slave-4tdj"
server_key: "a03bfd3cd127a4f3c59f9dc441ca5eaa"
server_key_iv: "f00adc1b1678db1c93e34559c3abe8c8"
server_key_version: "encryptionkey@0"
I20260806 06:26:46.318480 29509 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260806 06:26:46.319157 29531 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 06:26:46.319300 29509 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260806 06:26:46.319386 29509 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785997581049705-21554-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785997581049705-21554-0/minicluster-data/ts-0/wal
uuid: "3c557ebd102846e19d621703a58a7074"
format_stamp: "Formatted at 2026-08-06 06:26:46 on dist-test-slave-4tdj"
server_key: "a03bfd3cd127a4f3c59f9dc441ca5eaa"
server_key_iv: "f00adc1b1678db1c93e34559c3abe8c8"
server_key_version: "encryptionkey@0"
I20260806 06:26:46.319440 29509 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785997581049705-21554-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785997581049705-21554-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785997581049705-21554-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 06:26:46.331346 29509 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 06:26:46.333783 29509 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 06:26:46.333931 29509 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 06:26:46.334167 29509 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 06:26:46.334550 29509 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 06:26:46.334599 29509 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 06:26:46.334635 29509 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 06:26:46.334656 29509 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 06:26:46.342495 29509 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.129:43489
I20260806 06:26:46.342554 29705 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.129:43489 every 8 connection(s)
I20260806 06:26:46.342840 29509 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785997581049705-21554-0/minicluster-data/ts-0/data/info.pb
I20260806 06:26:46.347684 21554 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHhLhT/build/release/bin/kudu as pid 29509
I20260806 06:26:46.347797 21554 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785997581049705-21554-0/minicluster-data/ts-0/wal/instance
I20260806 06:26:46.347960 21554 external_mini_cluster.cc:1428] Setting key 8a11d716fb0d8ed9efb5b7ee6be07480
Aug 06 06:26:46 dist-test-slave-4tdj krb5kdc[29356](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997606, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.129@KRBTEST.COM for kudu/127.21.12.190@KRBTEST.COM
I20260806 06:26:46.351975 29499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:46.343610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.129:39847 (local address 127.21.12.190:35581)
0806 06:26:46.343669 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0806 06:26:46.343671 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:26:46.343853 (+ 182us) server_negotiation.cc:407] Connection header received
0806 06:26:46.344787 (+ 934us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:26:46.344788 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:26:46.344806 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:26:46.344854 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:26:46.345489 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:46.345992 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:46.346999 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:46.347212 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:46.347270 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:46.349314 (+ 2044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:26:46.349316 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:26:46.349316 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 06:26:46.349324 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:26:46.350718 (+ 1394us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:26:46.351104 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:26:46.351105 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:26:46.351106 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:26:46.351169 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:26:46.351375 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:26:46.351376 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:26:46.351376 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:26:46.351504 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:26:46.351569 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0806 06:26:46.351800 (+ 231us) server_negotiation.cc:299] Negotiation successful
0806 06:26:46.351889 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260806 06:26:46.352135 29708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:46.343581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:35581 (local address 127.21.12.129:39847)
0806 06:26:46.343734 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0806 06:26:46.343772 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0806 06:26:46.344680 (+ 908us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:26:46.344934 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:26:46.344938 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:26:46.345195 (+ 257us) client_negotiation.cc:189] Negotiated authn=SASL
0806 06:26:46.345398 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:46.345402 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:46.346089 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:46.346090 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:46.346917 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:46.346920 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:46.347284 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:46.347285 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:46.347397 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:46.347900 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 06:26:46.347912 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 06:26:46.349223 (+ 1311us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 06:26:46.350806 (+ 1583us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:26:46.350812 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:26:46.350814 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:26:46.351030 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:26:46.351231 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:26:46.351231 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:26:46.351232 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:26:46.351322 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:26:46.351566 (+ 244us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 06:26:46.351569 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 06:26:46.351687 (+ 118us) client_negotiation.cc:769] Sending connection context
0806 06:26:46.351763 (+ 76us) client_negotiation.cc:240] Negotiation successful
0806 06:26:46.351913 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":72,"threads_started":1}
I20260806 06:26:46.352648 29706 heartbeater.cc:347] Connected to a master server at 127.21.12.190:35581
I20260806 06:26:46.352759 29706 heartbeater.cc:464] Registering TS with master...
I20260806 06:26:46.352953 29706 heartbeater.cc:511] Master 127.21.12.190:35581 requested a full tablet report, sending...
I20260806 06:26:46.353467 29415 ts_manager.cc:194] Registered new tserver with Master: 3c557ebd102846e19d621703a58a7074 (127.21.12.129:43489)
I20260806 06:26:46.354056 29415 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.12.129@KRBTEST.COM'} at 127.21.12.129:39847
WARNING: no policy specified for kudu/127.21.12.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.12.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.12.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.130.keytab.
Entry for principal kudu/127.21.12.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.130.keytab.
I20260806 06:26:46.373224 21554 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHhLhT/build/release/bin/kudu
/tmp/dist-test-taskFHhLhT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785997581049705-21554-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785997581049705-21554-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785997581049705-21554-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785997581049705-21554-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.12.130:0
--local_ip_for_outbound_sockets=127.21.12.130
--webserver_interface=127.21.12.130
--webserver_port=0
--tserver_master_addrs=127.21.12.190:35581
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.130.keytab
--principal=kudu/127.21.12.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.148:44239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785997581049705-21554-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785997581049705-21554-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785997581049705-21554-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 06:26:46.453614 29711 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 06:26:46.453810 29711 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 06:26:46.453843 29711 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 06:26:46.454998 29711 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 06:26:46.455039 29711 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 06:26:46.455073 29711 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.12.130
I20260806 06:26:46.456518 29711 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.148:44239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785997581049705-21554-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785997581049705-21554-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.130.keytab
--principal=kudu/127.21.12.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.12.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785997581049705-21554-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.12.130
--webserver_port=0
--tserver_master_addrs=127.21.12.190:35581
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29711
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.12.130
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785997581049705-21554-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 05:11:02 UTC on ccycloud.quasar-hvzbbz.root.comops.site
build id 4569617
I20260806 06:26:46.456740 29711 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 06:26:46.456949 29711 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 06:26:46.461881 29723 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:26:46.461853 29722 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:26:46.461841 29725 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:26:46.462078 29711 server_base.cc:1034] running on GCE node
I20260806 06:26:46.462280 29711 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 06:26:46.462527 29711 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 06:26:46.463670 29711 hybrid_clock.cc:648] HybridClock initialized: now 1785997606463657 us; error 29 us; skew 500 ppm
Aug 06 06:26:46 dist-test-slave-4tdj krb5kdc[29356](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997606, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 06:26:46.466068 29711 init.cc:377] Logged in from keytab as kudu/127.21.12.130@KRBTEST.COM (short username kudu)
I20260806 06:26:46.466521 29711 webserver.cc:460] Webserver started at http://127.21.12.130:33491/ using document root <none> and password file <none>
I20260806 06:26:46.466747 29711 fs_manager.cc:359] Metadata directory not provided
I20260806 06:26:46.466790 29711 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 06:26:46.466914 29711 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 06:26:46.467901 29711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785997581049705-21554-0/minicluster-data/ts-1/data/instance:
uuid: "8468096bd137478f9e53e514b5afb290"
format_stamp: "Formatted at 2026-08-06 06:26:46 on dist-test-slave-4tdj"
server_key: "2b65773f894917e90c40c605317d6c76"
server_key_iv: "314d975db9709c81889412c64d81e70d"
server_key_version: "encryptionkey@0"
I20260806 06:26:46.468222 29711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785997581049705-21554-0/minicluster-data/ts-1/wal/instance:
uuid: "8468096bd137478f9e53e514b5afb290"
format_stamp: "Formatted at 2026-08-06 06:26:46 on dist-test-slave-4tdj"
server_key: "2b65773f894917e90c40c605317d6c76"
server_key_iv: "314d975db9709c81889412c64d81e70d"
server_key_version: "encryptionkey@0"
I20260806 06:26:46.469447 29711 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260806 06:26:46.470141 29734 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 06:26:46.470269 29711 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260806 06:26:46.470386 29711 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785997581049705-21554-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785997581049705-21554-0/minicluster-data/ts-1/wal
uuid: "8468096bd137478f9e53e514b5afb290"
format_stamp: "Formatted at 2026-08-06 06:26:46 on dist-test-slave-4tdj"
server_key: "2b65773f894917e90c40c605317d6c76"
server_key_iv: "314d975db9709c81889412c64d81e70d"
server_key_version: "encryptionkey@0"
I20260806 06:26:46.470453 29711 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785997581049705-21554-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785997581049705-21554-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785997581049705-21554-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 06:26:46.483875 29711 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 06:26:46.485713 29711 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 06:26:46.485828 29711 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 06:26:46.486047 29711 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 06:26:46.486366 29711 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 06:26:46.486405 29711 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 06:26:46.486424 29711 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 06:26:46.486438 29711 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 06:26:46.491909 29711 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.130:35029
I20260806 06:26:46.491966 29904 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.130:35029 every 8 connection(s)
I20260806 06:26:46.492244 29711 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785997581049705-21554-0/minicluster-data/ts-1/data/info.pb
I20260806 06:26:46.497735 21554 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHhLhT/build/release/bin/kudu as pid 29711
I20260806 06:26:46.497840 21554 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785997581049705-21554-0/minicluster-data/ts-1/wal/instance
I20260806 06:26:46.498037 21554 external_mini_cluster.cc:1428] Setting key 014f5d15a3633dc3266aec2f1b57465c
Aug 06 06:26:46 dist-test-slave-4tdj krb5kdc[29356](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997606, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.130@KRBTEST.COM for kudu/127.21.12.190@KRBTEST.COM
I20260806 06:26:46.501390 29499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:46.492930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.130:52201 (local address 127.21.12.190:35581)
0806 06:26:46.492995 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0806 06:26:46.492997 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:26:46.493155 (+ 158us) server_negotiation.cc:407] Connection header received
0806 06:26:46.494069 (+ 914us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:26:46.494070 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:26:46.494082 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:26:46.494137 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:26:46.494727 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:46.495274 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:46.496469 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:46.496674 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:46.496730 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:46.498912 (+ 2182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:26:46.498913 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:26:46.498914 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 06:26:46.498922 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:26:46.500134 (+ 1212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:26:46.500497 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:26:46.500499 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:26:46.500499 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:26:46.500559 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:26:46.500774 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:26:46.500775 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:26:46.500775 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:26:46.500880 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:26:46.500934 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0806 06:26:46.501197 (+ 263us) server_negotiation.cc:299] Negotiation successful
0806 06:26:46.501300 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260806 06:26:46.501525 29907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:46.492890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:35581 (local address 127.21.12.130:52201)
0806 06:26:46.493040 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0806 06:26:46.493080 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0806 06:26:46.493959 (+ 879us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:26:46.494191 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:26:46.494194 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:26:46.494471 (+ 277us) client_negotiation.cc:189] Negotiated authn=SASL
0806 06:26:46.494660 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:46.494665 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:46.495370 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:46.495371 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:46.496386 (+ 1015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:46.496389 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:46.496744 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:46.496745 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:46.496866 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:46.497260 (+ 394us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 06:26:46.497267 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0806 06:26:46.498826 (+ 1559us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 06:26:46.500210 (+ 1384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:26:46.500217 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:26:46.500219 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:26:46.500428 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:26:46.500613 (+ 185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:26:46.500613 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:26:46.500614 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:26:46.500719 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:26:46.500933 (+ 214us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 06:26:46.500940 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 06:26:46.501078 (+ 138us) client_negotiation.cc:769] Sending connection context
0806 06:26:46.501165 (+ 87us) client_negotiation.cc:240] Negotiation successful
0806 06:26:46.501320 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":79,"threads_started":1}
I20260806 06:26:46.501922 29905 heartbeater.cc:347] Connected to a master server at 127.21.12.190:35581
I20260806 06:26:46.502029 29905 heartbeater.cc:464] Registering TS with master...
I20260806 06:26:46.502206 29905 heartbeater.cc:511] Master 127.21.12.190:35581 requested a full tablet report, sending...
I20260806 06:26:46.502661 29415 ts_manager.cc:194] Registered new tserver with Master: 8468096bd137478f9e53e514b5afb290 (127.21.12.130:35029)
I20260806 06:26:46.503156 29415 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.12.130@KRBTEST.COM'} at 127.21.12.130:52201
WARNING: no policy specified for kudu/127.21.12.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.12.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.12.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.131.keytab.
Entry for principal kudu/127.21.12.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.131.keytab.
I20260806 06:26:46.523123 21554 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHhLhT/build/release/bin/kudu
/tmp/dist-test-taskFHhLhT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785997581049705-21554-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785997581049705-21554-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785997581049705-21554-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785997581049705-21554-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.12.131:0
--local_ip_for_outbound_sockets=127.21.12.131
--webserver_interface=127.21.12.131
--webserver_port=0
--tserver_master_addrs=127.21.12.190:35581
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.131.keytab
--principal=kudu/127.21.12.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.148:44239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785997581049705-21554-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785997581049705-21554-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785997581049705-21554-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 06:26:46.602908 29917 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 06:26:46.603106 29917 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 06:26:46.603128 29917 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 06:26:46.604208 29917 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 06:26:46.604240 29917 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 06:26:46.604264 29917 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.12.131
I20260806 06:26:46.605691 29917 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.148:44239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785997581049705-21554-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785997581049705-21554-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.131.keytab
--principal=kudu/127.21.12.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.12.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785997581049705-21554-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.12.131
--webserver_port=0
--tserver_master_addrs=127.21.12.190:35581
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29917
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.12.131
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785997581049705-21554-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 05:11:02 UTC on ccycloud.quasar-hvzbbz.root.comops.site
build id 4569617
I20260806 06:26:46.605880 29917 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 06:26:46.606065 29917 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 06:26:46.610833 29929 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:26:46.610847 29927 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:26:46.610853 29925 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:26:46.610963 29917 server_base.cc:1034] running on GCE node
I20260806 06:26:46.611241 29917 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 06:26:46.611443 29917 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 06:26:46.612581 29917 hybrid_clock.cc:648] HybridClock initialized: now 1785997606612559 us; error 34 us; skew 500 ppm
Aug 06 06:26:46 dist-test-slave-4tdj krb5kdc[29356](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997606, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 06:26:46.614931 29917 init.cc:377] Logged in from keytab as kudu/127.21.12.131@KRBTEST.COM (short username kudu)
I20260806 06:26:46.615351 29917 webserver.cc:460] Webserver started at http://127.21.12.131:41289/ using document root <none> and password file <none>
I20260806 06:26:46.615563 29917 fs_manager.cc:359] Metadata directory not provided
I20260806 06:26:46.615610 29917 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 06:26:46.615736 29917 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 06:26:46.617034 29917 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785997581049705-21554-0/minicluster-data/ts-2/data/instance:
uuid: "de22b9ca53e44cf59ee353a0690113b6"
format_stamp: "Formatted at 2026-08-06 06:26:46 on dist-test-slave-4tdj"
server_key: "c5e50cb34e82d0589fac45d03e207ace"
server_key_iv: "c15695101efdb754667951363892688b"
server_key_version: "encryptionkey@0"
I20260806 06:26:46.617347 29917 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785997581049705-21554-0/minicluster-data/ts-2/wal/instance:
uuid: "de22b9ca53e44cf59ee353a0690113b6"
format_stamp: "Formatted at 2026-08-06 06:26:46 on dist-test-slave-4tdj"
server_key: "c5e50cb34e82d0589fac45d03e207ace"
server_key_iv: "c15695101efdb754667951363892688b"
server_key_version: "encryptionkey@0"
I20260806 06:26:46.618558 29917 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260806 06:26:46.619189 29937 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 06:26:46.619318 29917 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260806 06:26:46.619402 29917 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785997581049705-21554-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785997581049705-21554-0/minicluster-data/ts-2/wal
uuid: "de22b9ca53e44cf59ee353a0690113b6"
format_stamp: "Formatted at 2026-08-06 06:26:46 on dist-test-slave-4tdj"
server_key: "c5e50cb34e82d0589fac45d03e207ace"
server_key_iv: "c15695101efdb754667951363892688b"
server_key_version: "encryptionkey@0"
I20260806 06:26:46.619457 29917 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785997581049705-21554-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785997581049705-21554-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785997581049705-21554-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 06:26:46.631862 29917 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 06:26:46.633512 29917 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 06:26:46.633641 29917 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 06:26:46.633828 29917 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 06:26:46.634107 29917 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 06:26:46.634141 29917 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 06:26:46.634168 29917 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 06:26:46.634192 29917 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 06:26:46.640102 29917 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.131:42985
I20260806 06:26:46.640157 30111 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.131:42985 every 8 connection(s)
I20260806 06:26:46.640448 29917 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785997581049705-21554-0/minicluster-data/ts-2/data/info.pb
Aug 06 06:26:46 dist-test-slave-4tdj krb5kdc[29356](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997606, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.131@KRBTEST.COM for kudu/127.21.12.190@KRBTEST.COM
I20260806 06:26:46.647954 21554 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHhLhT/build/release/bin/kudu as pid 29917
I20260806 06:26:46.648046 21554 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785997581049705-21554-0/minicluster-data/ts-2/wal/instance
I20260806 06:26:46.648232 21554 external_mini_cluster.cc:1428] Setting key efcf269964a8fa72b5866ffa140a50e4
I20260806 06:26:46.649396 29499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:46.641114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.131:45411 (local address 127.21.12.190:35581)
0806 06:26:46.641160 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0806 06:26:46.641161 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0806 06:26:46.641398 (+ 237us) server_negotiation.cc:407] Connection header received
0806 06:26:46.642324 (+ 926us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:26:46.642325 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:26:46.642339 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:26:46.642386 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:26:46.643005 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:46.643497 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:46.644496 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:46.644685 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:46.644742 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:46.646835 (+ 2093us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:26:46.646836 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:26:46.646837 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 06:26:46.646845 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:26:46.648030 (+ 1185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:26:46.648454 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:26:46.648456 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:26:46.648456 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:26:46.648526 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:26:46.648776 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:26:46.648778 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:26:46.648778 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:26:46.648908 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:26:46.648967 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0806 06:26:46.649229 (+ 262us) server_negotiation.cc:299] Negotiation successful
0806 06:26:46.649311 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260806 06:26:46.649536 30116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:46.641121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:35581 (local address 127.21.12.131:45411)
0806 06:26:46.641281 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0806 06:26:46.641315 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0806 06:26:46.642189 (+ 874us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:26:46.642467 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:26:46.642472 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:26:46.642730 (+ 258us) client_negotiation.cc:189] Negotiated authn=SASL
0806 06:26:46.642915 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:46.642919 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:46.643579 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:46.643580 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:46.644373 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:46.644376 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:46.644757 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:46.644758 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:46.644871 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:46.645327 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 06:26:46.645336 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0806 06:26:46.646741 (+ 1405us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 06:26:46.648121 (+ 1380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:26:46.648128 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:26:46.648130 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:26:46.648383 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:26:46.648590 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:26:46.648591 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:26:46.648591 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:26:46.648717 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:26:46.648967 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 06:26:46.648974 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 06:26:46.649110 (+ 136us) client_negotiation.cc:769] Sending connection context
0806 06:26:46.649194 (+ 84us) client_negotiation.cc:240] Negotiation successful
0806 06:26:46.649336 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":79,"threads_started":1}
I20260806 06:26:46.649923 30112 heartbeater.cc:347] Connected to a master server at 127.21.12.190:35581
I20260806 06:26:46.650014 30112 heartbeater.cc:464] Registering TS with master...
I20260806 06:26:46.650180 30112 heartbeater.cc:511] Master 127.21.12.190:35581 requested a full tablet report, sending...
I20260806 06:26:46.650547 29415 ts_manager.cc:194] Registered new tserver with Master: de22b9ca53e44cf59ee353a0690113b6 (127.21.12.131:42985)
I20260806 06:26:46.650875 29415 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.12.131@KRBTEST.COM'} at 127.21.12.131:45411
I20260806 06:26:46.659850 21554 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785997581049705-21554-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785997581049705-21554-0/krb5kdc/test-admin.keytab.
Aug 06 06:26:46 dist-test-slave-4tdj krb5kdc[29356](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997606, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 06:26:46.675149 21554 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260806 06:26:46.676787 30123 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 06:26:46.675961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54290 (local address 127.21.12.129:43489)
0806 06:26:46.676123 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0806 06:26:46.676127 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 06:26:46.676136 (+ 9us) server_negotiation.cc:407] Connection header received
0806 06:26:46.676161 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:26:46.676163 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:26:46.676176 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:26:46.676238 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:26:46.676706 (+ 468us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54290: BlockingRecv error: recv got EOF from 127.0.0.1:54290 (error 108)
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":77,"threads_started":1}
W20260806 06:26:46.676858 21554 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785997581049705-21554-0/krb5kdc/krb5cc' of type FILE
Aug 06 06:26:46 dist-test-slave-4tdj krb5kdc[29356](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997606, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 06:26:46.677774 21554 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 06 06:26:46 dist-test-slave-4tdj krb5kdc[29356](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997606, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 06:26:46.678645 21554 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 06 06:26:46 dist-test-slave-4tdj krb5kdc[29356](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997606, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.12.129@KRBTEST.COM
I20260806 06:26:46.686034 30123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:46.679117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54296 (local address 127.21.12.129:43489)
0806 06:26:46.679165 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0806 06:26:46.679167 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:26:46.679234 (+ 67us) server_negotiation.cc:407] Connection header received
0806 06:26:46.679281 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:26:46.679282 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:26:46.679291 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:26:46.679337 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:26:46.679710 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:46.680418 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:46.681167 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:46.681485 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:46.681551 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:46.683593 (+ 2042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:26:46.683601 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:26:46.683601 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 06:26:46.683608 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:26:46.685009 (+ 1401us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:26:46.685323 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:26:46.685326 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:26:46.685326 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:26:46.685374 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:26:46.685501 (+ 127us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:26:46.685501 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:26:46.685502 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:26:46.685638 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:26:46.685741 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0806 06:26:46.685869 (+ 128us) server_negotiation.cc:299] Negotiation successful
0806 06:26:46.685930 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
W20260806 06:26:46.687813 30123 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 06:26:46.687328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54298 (local address 127.21.12.129:43489)
0806 06:26:46.687363 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0806 06:26:46.687365 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:26:46.687394 (+ 29us) server_negotiation.cc:407] Connection header received
0806 06:26:46.687455 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:26:46.687455 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:26:46.687464 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:26:46.687502 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:26:46.687774 (+ 272us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54298: BlockingRecv error: recv got EOF from 127.0.0.1:54298 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260806 06:26:46.687891 21554 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785997581049705-21554-0/krb5kdc/krb5cc' of type FILE
Aug 06 06:26:46 dist-test-slave-4tdj krb5kdc[29356](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997606, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 06:26:46.688762 21554 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 06 06:26:46 dist-test-slave-4tdj krb5kdc[29356](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997606, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 06:26:46.689517 21554 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 06 06:26:46 dist-test-slave-4tdj krb5kdc[29356](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997606, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.12.129@KRBTEST.COM
I20260806 06:26:46.696055 30123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:46.690039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54302 (local address 127.21.12.129:43489)
0806 06:26:46.690088 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0806 06:26:46.690090 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:26:46.690103 (+ 13us) server_negotiation.cc:407] Connection header received
0806 06:26:46.690172 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:26:46.690173 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:26:46.690184 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:26:46.690231 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:26:46.690607 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:46.691082 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:46.691736 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:46.691926 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:46.691977 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:46.693956 (+ 1979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:26:46.693957 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:26:46.693958 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 06:26:46.693965 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:26:46.695203 (+ 1238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:26:46.695512 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:26:46.695513 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:26:46.695514 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:26:46.695577 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:26:46.695712 (+ 135us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:26:46.695713 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:26:46.695713 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:26:46.695790 (+ 77us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:26:46.695852 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0806 06:26:46.695923 (+ 71us) server_negotiation.cc:299] Negotiation successful
0806 06:26:46.695976 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
W20260806 06:26:46.697378 30123 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 06:26:46.696958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54316 (local address 127.21.12.129:43489)
0806 06:26:46.697018 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0806 06:26:46.697019 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0806 06:26:46.697027 (+ 8us) server_negotiation.cc:407] Connection header received
0806 06:26:46.697074 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:26:46.697074 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:26:46.697082 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:26:46.697123 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:26:46.697336 (+ 213us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54316: BlockingRecv error: recv got EOF from 127.0.0.1:54316 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260806 06:26:46.697475 21554 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785997581049705-21554-0/krb5kdc/krb5cc' of type FILE
Aug 06 06:26:46 dist-test-slave-4tdj krb5kdc[29356](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997606, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 06:26:46.698346 21554 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 06 06:26:46 dist-test-slave-4tdj krb5kdc[29356](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997606, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 06:26:46.699076 21554 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 06 06:26:46 dist-test-slave-4tdj krb5kdc[29356](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997606, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.12.129@KRBTEST.COM
I20260806 06:26:46.705089 30123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:46.699501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54328 (local address 127.21.12.129:43489)
0806 06:26:46.699560 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0806 06:26:46.699561 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0806 06:26:46.699573 (+ 12us) server_negotiation.cc:407] Connection header received
0806 06:26:46.699628 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:26:46.699629 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:26:46.699636 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:26:46.699670 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:26:46.699953 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:46.700396 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:46.701016 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:46.701192 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:46.701237 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:46.703045 (+ 1808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:26:46.703046 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:26:46.703046 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 06:26:46.703053 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:26:46.704183 (+ 1130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:26:46.704466 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:26:46.704468 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:26:46.704468 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:26:46.704514 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:26:46.704676 (+ 162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:26:46.704677 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:26:46.704677 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:26:46.704761 (+ 84us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:26:46.704838 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0806 06:26:46.704937 (+ 99us) server_negotiation.cc:299] Negotiation successful
0806 06:26:46.705007 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260806 06:26:46.705667 21554 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHhLhT/build/release/bin/kudu with pid 29509
I20260806 06:26:46.716845 21554 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHhLhT/build/release/bin/kudu with pid 29711
I20260806 06:26:46.726472 21554 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHhLhT/build/release/bin/kudu with pid 29917
I20260806 06:26:46.734404 21554 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHhLhT/build/release/bin/kudu with pid 29374
2026-08-06T06:26:46Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (1968 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785997581049705-21554-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 06:26:46 dist-test-slave-4tdj krb5kdc[30146](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 06:26:46 dist-test-slave-4tdj krb5kdc[30146](info): set up 2 sockets
Aug 06 06:26:46 dist-test-slave-4tdj krb5kdc[30146](info): commencing operation
krb5kdc: starting...
W20260806 06:26:47.839717 21554 mini_kdc.cc:121] Time spent starting KDC: real 1.084s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785997581049705-21554-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785997581049705-21554-0/krb5kdc/test-user.keytab.
Aug 06 06:26:47 dist-test-slave-4tdj krb5kdc[30146](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997607, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T06:26:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T06:26:47Z Disabled control of system clock
WARNING: no policy specified for oryx/127.21.12.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.12.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.12.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785997581049705-21554-0/krb5kdc/oryx_127.21.12.190.keytab.
Entry for principal oryx/127.21.12.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785997581049705-21554-0/krb5kdc/oryx_127.21.12.190.keytab.
I20260806 06:26:47.953238 21554 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHhLhT/build/release/bin/kudu
/tmp/dist-test-taskFHhLhT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785997581049705-21554-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785997581049705-21554-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785997581049705-21554-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785997581049705-21554-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.12.190:41635
--webserver_interface=127.21.12.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785997581049705-21554-0/krb5kdc/oryx_127.21.12.190.keytab
--principal=oryx/127.21.12.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.148:39091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.12.190:41635
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785997581049705-21554-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785997581049705-21554-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785997581049705-21554-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 06:26:48.033023 30163 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 06:26:48.033228 30163 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 06:26:48.033248 30163 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 06:26:48.034386 30163 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 06:26:48.034425 30163 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 06:26:48.034437 30163 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 06:26:48.034449 30163 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 06:26:48.034459 30163 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 06:26:48.035884 30163 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.148:39091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785997581049705-21554-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785997581049705-21554-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.12.190:41635
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785997581049705-21554-0/krb5kdc/oryx_127.21.12.190.keytab
--principal=oryx/127.21.12.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.12.190:41635
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785997581049705-21554-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.12.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30163
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785997581049705-21554-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 05:11:02 UTC on ccycloud.quasar-hvzbbz.root.comops.site
build id 4569617
I20260806 06:26:48.036119 30163 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 06:26:48.036325 30163 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260806 06:26:48.041079 30163 server_base.cc:1034] running on GCE node
W20260806 06:26:48.041054 30174 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:26:48.041052 30173 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:26:48.041045 30176 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:26:48.041519 30163 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 06:26:48.041785 30163 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 06:26:48.042920 30163 hybrid_clock.cc:648] HybridClock initialized: now 1785997608042899 us; error 33 us; skew 500 ppm
Aug 06 06:26:48 dist-test-slave-4tdj krb5kdc[30146](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997608, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.12.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 06:26:48.045236 30163 init.cc:377] Logged in from keytab as oryx/127.21.12.190@KRBTEST.COM (short username oryx)
I20260806 06:26:48.045660 30163 webserver.cc:460] Webserver started at http://127.21.12.190:35183/ using document root <none> and password file <none>
I20260806 06:26:48.045849 30163 fs_manager.cc:359] Metadata directory not provided
I20260806 06:26:48.045894 30163 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 06:26:48.046016 30163 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 06:26:48.047031 30163 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785997581049705-21554-0/minicluster-data/master-0/data/instance:
uuid: "2589a7a56e404deda1060eb408ebf54c"
format_stamp: "Formatted at 2026-08-06 06:26:48 on dist-test-slave-4tdj"
server_key: "1350b528ad445ce81c17cdacb2c3dd7a"
server_key_iv: "b28ad15830f0cb8b2609f682695e83c7"
server_key_version: "encryptionkey@0"
I20260806 06:26:48.047374 30163 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785997581049705-21554-0/minicluster-data/master-0/wal/instance:
uuid: "2589a7a56e404deda1060eb408ebf54c"
format_stamp: "Formatted at 2026-08-06 06:26:48 on dist-test-slave-4tdj"
server_key: "1350b528ad445ce81c17cdacb2c3dd7a"
server_key_iv: "b28ad15830f0cb8b2609f682695e83c7"
server_key_version: "encryptionkey@0"
I20260806 06:26:48.048561 30163 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260806 06:26:48.049194 30185 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 06:26:48.049338 30163 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260806 06:26:48.049422 30163 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785997581049705-21554-0/minicluster-data/master-0/data,/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785997581049705-21554-0/minicluster-data/master-0/wal
uuid: "2589a7a56e404deda1060eb408ebf54c"
format_stamp: "Formatted at 2026-08-06 06:26:48 on dist-test-slave-4tdj"
server_key: "1350b528ad445ce81c17cdacb2c3dd7a"
server_key_iv: "b28ad15830f0cb8b2609f682695e83c7"
server_key_version: "encryptionkey@0"
I20260806 06:26:48.049476 30163 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785997581049705-21554-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785997581049705-21554-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785997581049705-21554-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 06:26:48.068545 30163 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 06:26:48.070199 30163 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 06:26:48.070359 30163 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 06:26:48.073908 30163 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.190:41635
I20260806 06:26:48.073944 30271 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.190:41635 every 8 connection(s)
I20260806 06:26:48.074317 30163 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785997581049705-21554-0/minicluster-data/master-0/data/info.pb
I20260806 06:26:48.075238 30272 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 06:26:48.077399 30272 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2589a7a56e404deda1060eb408ebf54c: Bootstrap starting.
I20260806 06:26:48.077658 21554 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHhLhT/build/release/bin/kudu as pid 30163
I20260806 06:26:48.077775 21554 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785997581049705-21554-0/minicluster-data/master-0/wal/instance
I20260806 06:26:48.077965 21554 external_mini_cluster.cc:1428] Setting key 397a9f02876e76c2363de78698e9f750
I20260806 06:26:48.078092 30272 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2589a7a56e404deda1060eb408ebf54c: Neither blocks nor log segments found. Creating new log.
I20260806 06:26:48.078406 30272 log.cc:824] T 00000000000000000000000000000000 P 2589a7a56e404deda1060eb408ebf54c: Log is configured to *not* fsync() on all Append() calls
I20260806 06:26:48.079147 30272 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2589a7a56e404deda1060eb408ebf54c: No bootstrap required, opened a new log
I20260806 06:26:48.081241 30272 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2589a7a56e404deda1060eb408ebf54c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2589a7a56e404deda1060eb408ebf54c" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 41635 } }
I20260806 06:26:48.081406 30272 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2589a7a56e404deda1060eb408ebf54c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 06:26:48.081439 30272 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2589a7a56e404deda1060eb408ebf54c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2589a7a56e404deda1060eb408ebf54c, State: Initialized, Role: FOLLOWER
I20260806 06:26:48.081568 30272 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2589a7a56e404deda1060eb408ebf54c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2589a7a56e404deda1060eb408ebf54c" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 41635 } }
I20260806 06:26:48.081653 30272 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2589a7a56e404deda1060eb408ebf54c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 06:26:48.081686 30272 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2589a7a56e404deda1060eb408ebf54c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 06:26:48.081727 30272 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2589a7a56e404deda1060eb408ebf54c [term 0 FOLLOWER]: Advancing to term 1
I20260806 06:26:48.082270 30272 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2589a7a56e404deda1060eb408ebf54c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2589a7a56e404deda1060eb408ebf54c" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 41635 } }
I20260806 06:26:48.082397 30272 leader_election.cc:302] T 00000000000000000000000000000000 P 2589a7a56e404deda1060eb408ebf54c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2589a7a56e404deda1060eb408ebf54c; no voters:
I20260806 06:26:48.082566 30272 leader_election.cc:288] T 00000000000000000000000000000000 P 2589a7a56e404deda1060eb408ebf54c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 06:26:48.082598 30282 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2589a7a56e404deda1060eb408ebf54c [term 1 FOLLOWER]: Leader election won for term 1
I20260806 06:26:48.082743 30282 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2589a7a56e404deda1060eb408ebf54c [term 1 LEADER]: Becoming Leader. State: Replica: 2589a7a56e404deda1060eb408ebf54c, State: Running, Role: LEADER
I20260806 06:26:48.082823 30272 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2589a7a56e404deda1060eb408ebf54c [sys.catalog]: configured and running, proceeding with master startup.
I20260806 06:26:48.082828 30282 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2589a7a56e404deda1060eb408ebf54c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2589a7a56e404deda1060eb408ebf54c" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 41635 } }
I20260806 06:26:48.083209 30286 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2589a7a56e404deda1060eb408ebf54c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2589a7a56e404deda1060eb408ebf54c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2589a7a56e404deda1060eb408ebf54c" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 41635 } } }
I20260806 06:26:48.083313 30286 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2589a7a56e404deda1060eb408ebf54c [sys.catalog]: This master's current role is: LEADER
I20260806 06:26:48.083204 30285 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2589a7a56e404deda1060eb408ebf54c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2589a7a56e404deda1060eb408ebf54c. Latest consensus state: current_term: 1 leader_uuid: "2589a7a56e404deda1060eb408ebf54c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2589a7a56e404deda1060eb408ebf54c" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 41635 } } }
I20260806 06:26:48.083491 30285 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2589a7a56e404deda1060eb408ebf54c [sys.catalog]: This master's current role is: LEADER
I20260806 06:26:48.083609 30292 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 06:26:48.084105 30292 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Aug 06 06:26:48 dist-test-slave-4tdj krb5kdc[30146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997607, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.21.12.190@KRBTEST.COM
I20260806 06:26:48.085994 30292 catalog_manager.cc:1347] Generated new cluster ID: 2793aaf3eaab49c4806dfa641e5153fa
I20260806 06:26:48.086045 30292 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 06:26:48.089038 30277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:48.078484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50698 (local address 127.21.12.190:41635)
0806 06:26:48.078667 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0806 06:26:48.078677 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 06:26:48.078711 (+ 34us) server_negotiation.cc:407] Connection header received
0806 06:26:48.079193 (+ 482us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:26:48.079205 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:26:48.079280 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:26:48.079391 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:26:48.080120 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:48.080945 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:48.081933 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:48.082382 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:48.082483 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:48.085247 (+ 2764us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:26:48.085251 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:26:48.085251 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 06:26:48.085264 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:26:48.087123 (+ 1859us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:26:48.087468 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:26:48.087471 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:26:48.087471 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:26:48.087561 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:26:48.087723 (+ 162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:26:48.087724 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:26:48.087724 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:26:48.087984 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:26:48.088126 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0806 06:26:48.088558 (+ 432us) server_negotiation.cc:299] Negotiation successful
0806 06:26:48.088683 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":80,"threads_started":1}
I20260806 06:26:48.093042 30292 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 06:26:48.093513 30292 catalog_manager.cc:1504] Loading token signing keys...
I20260806 06:26:48.101811 30292 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2589a7a56e404deda1060eb408ebf54c: Generated new TSK 0
I20260806 06:26:48.101950 30292 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.21.12.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.12.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.12.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785997581049705-21554-0/krb5kdc/oryx_127.21.12.129.keytab.
Entry for principal oryx/127.21.12.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785997581049705-21554-0/krb5kdc/oryx_127.21.12.129.keytab.
I20260806 06:26:48.128846 21554 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHhLhT/build/release/bin/kudu
/tmp/dist-test-taskFHhLhT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785997581049705-21554-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785997581049705-21554-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785997581049705-21554-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785997581049705-21554-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.12.129:0
--local_ip_for_outbound_sockets=127.21.12.129
--webserver_interface=127.21.12.129
--webserver_port=0
--tserver_master_addrs=127.21.12.190:41635
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785997581049705-21554-0/krb5kdc/oryx_127.21.12.129.keytab
--principal=oryx/127.21.12.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.148:39091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785997581049705-21554-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785997581049705-21554-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785997581049705-21554-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 06:26:48.207446 30313 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 06:26:48.207649 30313 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 06:26:48.207679 30313 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 06:26:48.208783 30313 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 06:26:48.208822 30313 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 06:26:48.208855 30313 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.12.129
I20260806 06:26:48.210255 30313 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.148:39091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785997581049705-21554-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785997581049705-21554-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785997581049705-21554-0/krb5kdc/oryx_127.21.12.129.keytab
--principal=oryx/127.21.12.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.12.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785997581049705-21554-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.12.129
--webserver_port=0
--tserver_master_addrs=127.21.12.190:41635
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30313
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.12.129
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785997581049705-21554-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 05:11:02 UTC on ccycloud.quasar-hvzbbz.root.comops.site
build id 4569617
I20260806 06:26:48.210497 30313 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 06:26:48.210703 30313 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 06:26:48.215273 30325 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:26:48.215281 30322 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:26:48.215322 30323 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:26:48.215621 30313 server_base.cc:1034] running on GCE node
I20260806 06:26:48.215811 30313 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 06:26:48.216012 30313 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 06:26:48.217150 30313 hybrid_clock.cc:648] HybridClock initialized: now 1785997608217133 us; error 28 us; skew 500 ppm
Aug 06 06:26:48 dist-test-slave-4tdj krb5kdc[30146](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997608, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.12.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 06:26:48.219528 30313 init.cc:377] Logged in from keytab as oryx/127.21.12.129@KRBTEST.COM (short username oryx)
I20260806 06:26:48.219950 30313 webserver.cc:460] Webserver started at http://127.21.12.129:35087/ using document root <none> and password file <none>
I20260806 06:26:48.220141 30313 fs_manager.cc:359] Metadata directory not provided
I20260806 06:26:48.220187 30313 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 06:26:48.220309 30313 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 06:26:48.221295 30313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785997581049705-21554-0/minicluster-data/ts-0/data/instance:
uuid: "9c272304769843f1af8f68c3d16ec185"
format_stamp: "Formatted at 2026-08-06 06:26:48 on dist-test-slave-4tdj"
server_key: "2fbaa09ac62a1ae18eb3b639f348d10f"
server_key_iv: "f116fe7437f0915b44ae77dfa9d5882b"
server_key_version: "encryptionkey@0"
I20260806 06:26:48.221624 30313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785997581049705-21554-0/minicluster-data/ts-0/wal/instance:
uuid: "9c272304769843f1af8f68c3d16ec185"
format_stamp: "Formatted at 2026-08-06 06:26:48 on dist-test-slave-4tdj"
server_key: "2fbaa09ac62a1ae18eb3b639f348d10f"
server_key_iv: "f116fe7437f0915b44ae77dfa9d5882b"
server_key_version: "encryptionkey@0"
I20260806 06:26:48.222894 30313 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260806 06:26:48.223542 30336 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 06:26:48.223693 30313 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260806 06:26:48.223775 30313 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785997581049705-21554-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785997581049705-21554-0/minicluster-data/ts-0/wal
uuid: "9c272304769843f1af8f68c3d16ec185"
format_stamp: "Formatted at 2026-08-06 06:26:48 on dist-test-slave-4tdj"
server_key: "2fbaa09ac62a1ae18eb3b639f348d10f"
server_key_iv: "f116fe7437f0915b44ae77dfa9d5882b"
server_key_version: "encryptionkey@0"
I20260806 06:26:48.223832 30313 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785997581049705-21554-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785997581049705-21554-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785997581049705-21554-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 06:26:48.244416 30313 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 06:26:48.246141 30313 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 06:26:48.246269 30313 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 06:26:48.246501 30313 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 06:26:48.246788 30313 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 06:26:48.246821 30313 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 06:26:48.246850 30313 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 06:26:48.246867 30313 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 06:26:48.252403 30313 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.129:41895
I20260806 06:26:48.252442 30521 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.129:41895 every 8 connection(s)
I20260806 06:26:48.252731 30313 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785997581049705-21554-0/minicluster-data/ts-0/data/info.pb
I20260806 06:26:48.253604 21554 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHhLhT/build/release/bin/kudu as pid 30313
I20260806 06:26:48.253718 21554 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785997581049705-21554-0/minicluster-data/ts-0/wal/instance
I20260806 06:26:48.253909 21554 external_mini_cluster.cc:1428] Setting key 05908ab0ec0030cba4999c13d962fb25
Aug 06 06:26:48 dist-test-slave-4tdj krb5kdc[30146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997608, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.12.129@KRBTEST.COM for oryx/127.21.12.190@KRBTEST.COM
I20260806 06:26:48.262387 30277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:48.253398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.129:56277 (local address 127.21.12.190:41635)
0806 06:26:48.253471 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0806 06:26:48.253474 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 06:26:48.253700 (+ 226us) server_negotiation.cc:407] Connection header received
0806 06:26:48.254326 (+ 626us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:26:48.254328 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:26:48.254339 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:26:48.254381 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:26:48.255514 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:48.256163 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:48.257131 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:48.257435 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:48.257511 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:48.259823 (+ 2312us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:26:48.259825 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:26:48.259825 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 06:26:48.259831 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:26:48.261037 (+ 1206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:26:48.261477 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:26:48.261478 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:26:48.261478 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:26:48.261541 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:26:48.261803 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:26:48.261803 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:26:48.261803 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:26:48.261888 (+ 85us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:26:48.261929 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0806 06:26:48.262232 (+ 303us) server_negotiation.cc:299] Negotiation successful
0806 06:26:48.262318 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260806 06:26:48.262681 30525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:48.253378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:41635 (local address 127.21.12.129:56277)
0806 06:26:48.253555 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0806 06:26:48.253602 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0806 06:26:48.254208 (+ 606us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:26:48.254460 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:26:48.254464 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:26:48.255166 (+ 702us) client_negotiation.cc:189] Negotiated authn=SASL
0806 06:26:48.255430 (+ 264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:48.255436 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:48.256272 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:48.256273 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:48.257050 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:48.257052 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:48.257522 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:48.257523 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:48.257666 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:48.258085 (+ 419us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 06:26:48.258094 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0806 06:26:48.259724 (+ 1630us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 06:26:48.261116 (+ 1392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:26:48.261124 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:26:48.261126 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:26:48.261409 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:26:48.261607 (+ 198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:26:48.261607 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:26:48.261608 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:26:48.261746 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:26:48.261938 (+ 192us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 06:26:48.261944 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 06:26:48.262098 (+ 154us) client_negotiation.cc:769] Sending connection context
0806 06:26:48.262214 (+ 116us) client_negotiation.cc:240] Negotiation successful
0806 06:26:48.262396 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":93,"threads_started":1}
WARNING: no policy specified for oryx/127.21.12.130@KRBTEST.COM; defaulting to no policy
I20260806 06:26:48.263170 30523 heartbeater.cc:347] Connected to a master server at 127.21.12.190:41635
I20260806 06:26:48.263252 30523 heartbeater.cc:464] Registering TS with master...
I20260806 06:26:48.263406 30523 heartbeater.cc:511] Master 127.21.12.190:41635 requested a full tablet report, sending...
I20260806 06:26:48.263856 30217 ts_manager.cc:194] Registered new tserver with Master: 9c272304769843f1af8f68c3d16ec185 (127.21.12.129:41895)
I20260806 06:26:48.264446 30217 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.21.12.129@KRBTEST.COM'} at 127.21.12.129:56277
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.12.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.12.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785997581049705-21554-0/krb5kdc/oryx_127.21.12.130.keytab.
Entry for principal oryx/127.21.12.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785997581049705-21554-0/krb5kdc/oryx_127.21.12.130.keytab.
I20260806 06:26:48.278916 21554 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHhLhT/build/release/bin/kudu
/tmp/dist-test-taskFHhLhT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785997581049705-21554-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785997581049705-21554-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785997581049705-21554-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785997581049705-21554-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.12.130:0
--local_ip_for_outbound_sockets=127.21.12.130
--webserver_interface=127.21.12.130
--webserver_port=0
--tserver_master_addrs=127.21.12.190:41635
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785997581049705-21554-0/krb5kdc/oryx_127.21.12.130.keytab
--principal=oryx/127.21.12.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.148:39091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785997581049705-21554-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785997581049705-21554-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785997581049705-21554-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 06:26:48.357263 30529 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 06:26:48.357446 30529 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 06:26:48.357475 30529 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 06:26:48.358580 30529 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 06:26:48.358618 30529 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 06:26:48.358650 30529 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.12.130
I20260806 06:26:48.360021 30529 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.148:39091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785997581049705-21554-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785997581049705-21554-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785997581049705-21554-0/krb5kdc/oryx_127.21.12.130.keytab
--principal=oryx/127.21.12.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.12.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785997581049705-21554-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.12.130
--webserver_port=0
--tserver_master_addrs=127.21.12.190:41635
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30529
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.12.130
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785997581049705-21554-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 05:11:02 UTC on ccycloud.quasar-hvzbbz.root.comops.site
build id 4569617
I20260806 06:26:48.360249 30529 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 06:26:48.360442 30529 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 06:26:48.365144 30536 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:26:48.365150 30534 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:26:48.365298 30529 server_base.cc:1034] running on GCE node
W20260806 06:26:48.365150 30539 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:26:48.365528 30529 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 06:26:48.365737 30529 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 06:26:48.366884 30529 hybrid_clock.cc:648] HybridClock initialized: now 1785997608366872 us; error 28 us; skew 500 ppm
Aug 06 06:26:48 dist-test-slave-4tdj krb5kdc[30146](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997608, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.12.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 06:26:48.369189 30529 init.cc:377] Logged in from keytab as oryx/127.21.12.130@KRBTEST.COM (short username oryx)
I20260806 06:26:48.369593 30529 webserver.cc:460] Webserver started at http://127.21.12.130:41827/ using document root <none> and password file <none>
I20260806 06:26:48.369776 30529 fs_manager.cc:359] Metadata directory not provided
I20260806 06:26:48.369819 30529 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 06:26:48.369938 30529 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 06:26:48.370926 30529 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785997581049705-21554-0/minicluster-data/ts-1/data/instance:
uuid: "08f0835258af4cfc8d2b541884f9ceba"
format_stamp: "Formatted at 2026-08-06 06:26:48 on dist-test-slave-4tdj"
server_key: "3794ee1678c56777f2a7ab621251d0d1"
server_key_iv: "239ef81d6d4e49e60f52bdbb4c5177d7"
server_key_version: "encryptionkey@0"
I20260806 06:26:48.371253 30529 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785997581049705-21554-0/minicluster-data/ts-1/wal/instance:
uuid: "08f0835258af4cfc8d2b541884f9ceba"
format_stamp: "Formatted at 2026-08-06 06:26:48 on dist-test-slave-4tdj"
server_key: "3794ee1678c56777f2a7ab621251d0d1"
server_key_iv: "239ef81d6d4e49e60f52bdbb4c5177d7"
server_key_version: "encryptionkey@0"
I20260806 06:26:48.372419 30529 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260806 06:26:48.373083 30550 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 06:26:48.373224 30529 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260806 06:26:48.373303 30529 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785997581049705-21554-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785997581049705-21554-0/minicluster-data/ts-1/wal
uuid: "08f0835258af4cfc8d2b541884f9ceba"
format_stamp: "Formatted at 2026-08-06 06:26:48 on dist-test-slave-4tdj"
server_key: "3794ee1678c56777f2a7ab621251d0d1"
server_key_iv: "239ef81d6d4e49e60f52bdbb4c5177d7"
server_key_version: "encryptionkey@0"
I20260806 06:26:48.373365 30529 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785997581049705-21554-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785997581049705-21554-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785997581049705-21554-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 06:26:48.391757 30529 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 06:26:48.393743 30529 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 06:26:48.393869 30529 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 06:26:48.394065 30529 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 06:26:48.394397 30529 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 06:26:48.394433 30529 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 06:26:48.394459 30529 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 06:26:48.394484 30529 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 06:26:48.399884 30529 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.130:45263
I20260806 06:26:48.399928 30724 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.130:45263 every 8 connection(s)
I20260806 06:26:48.400199 30529 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785997581049705-21554-0/minicluster-data/ts-1/data/info.pb
I20260806 06:26:48.403173 21554 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHhLhT/build/release/bin/kudu as pid 30529
I20260806 06:26:48.403265 21554 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785997581049705-21554-0/minicluster-data/ts-1/wal/instance
I20260806 06:26:48.403437 21554 external_mini_cluster.cc:1428] Setting key 1dbec43c52ef4d5dd88d8148387bfafb
Aug 06 06:26:48 dist-test-slave-4tdj krb5kdc[30146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997608, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.12.130@KRBTEST.COM for oryx/127.21.12.190@KRBTEST.COM
I20260806 06:26:48.409132 30277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:48.400814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.130:44181 (local address 127.21.12.190:41635)
0806 06:26:48.400869 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0806 06:26:48.400871 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:26:48.401049 (+ 178us) server_negotiation.cc:407] Connection header received
0806 06:26:48.401557 (+ 508us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:26:48.401559 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:26:48.401573 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:26:48.401642 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:26:48.402581 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:48.403075 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:48.404161 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:48.404366 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:48.404420 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:48.406366 (+ 1946us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:26:48.406368 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:26:48.406368 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 06:26:48.406376 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:26:48.407804 (+ 1428us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:26:48.408218 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:26:48.408219 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:26:48.408219 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:26:48.408286 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:26:48.408515 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:26:48.408516 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:26:48.408516 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:26:48.408660 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:26:48.408725 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0806 06:26:48.408949 (+ 224us) server_negotiation.cc:299] Negotiation successful
0806 06:26:48.409035 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260806 06:26:48.409189 30728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:48.400774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:41635 (local address 127.21.12.130:44181)
0806 06:26:48.400908 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0806 06:26:48.400952 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0806 06:26:48.401421 (+ 469us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:26:48.401724 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:26:48.401728 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:26:48.402272 (+ 544us) client_negotiation.cc:189] Negotiated authn=SASL
0806 06:26:48.402491 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:48.402496 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:48.403157 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:48.403158 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:48.404092 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:48.404095 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:48.404432 (+ 337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:48.404432 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:48.404555 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:48.404978 (+ 423us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 06:26:48.404987 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0806 06:26:48.406265 (+ 1278us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 06:26:48.407896 (+ 1631us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:26:48.407902 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:26:48.407904 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:26:48.408134 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:26:48.408353 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:26:48.408353 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:26:48.408354 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:26:48.408449 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:26:48.408717 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 06:26:48.408720 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 06:26:48.408831 (+ 111us) client_negotiation.cc:769] Sending connection context
0806 06:26:48.408906 (+ 75us) client_negotiation.cc:240] Negotiation successful
0806 06:26:48.409029 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":62,"threads_started":1}
I20260806 06:26:48.409646 30725 heartbeater.cc:347] Connected to a master server at 127.21.12.190:41635
I20260806 06:26:48.409729 30725 heartbeater.cc:464] Registering TS with master...
I20260806 06:26:48.409890 30725 heartbeater.cc:511] Master 127.21.12.190:41635 requested a full tablet report, sending...
I20260806 06:26:48.410259 30217 ts_manager.cc:194] Registered new tserver with Master: 08f0835258af4cfc8d2b541884f9ceba (127.21.12.130:45263)
I20260806 06:26:48.410782 30217 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.21.12.130@KRBTEST.COM'} at 127.21.12.130:44181
WARNING: no policy specified for oryx/127.21.12.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.12.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.12.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785997581049705-21554-0/krb5kdc/oryx_127.21.12.131.keytab.
Entry for principal oryx/127.21.12.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785997581049705-21554-0/krb5kdc/oryx_127.21.12.131.keytab.
I20260806 06:26:48.429152 21554 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHhLhT/build/release/bin/kudu
/tmp/dist-test-taskFHhLhT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785997581049705-21554-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785997581049705-21554-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785997581049705-21554-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785997581049705-21554-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.12.131:0
--local_ip_for_outbound_sockets=127.21.12.131
--webserver_interface=127.21.12.131
--webserver_port=0
--tserver_master_addrs=127.21.12.190:41635
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785997581049705-21554-0/krb5kdc/oryx_127.21.12.131.keytab
--principal=oryx/127.21.12.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.148:39091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785997581049705-21554-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785997581049705-21554-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785997581049705-21554-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 06:26:48.509869 30731 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 06:26:48.510061 30731 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 06:26:48.510090 30731 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 06:26:48.511204 30731 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 06:26:48.511243 30731 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 06:26:48.511277 30731 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.12.131
I20260806 06:26:48.512692 30731 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.148:39091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785997581049705-21554-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785997581049705-21554-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785997581049705-21554-0/krb5kdc/oryx_127.21.12.131.keytab
--principal=oryx/127.21.12.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.12.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785997581049705-21554-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.12.131
--webserver_port=0
--tserver_master_addrs=127.21.12.190:41635
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30731
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.12.131
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785997581049705-21554-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 05:11:02 UTC on ccycloud.quasar-hvzbbz.root.comops.site
build id 4569617
I20260806 06:26:48.512921 30731 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 06:26:48.513123 30731 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 06:26:48.517915 30743 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:26:48.517927 30748 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:26:48.517967 30731 server_base.cc:1034] running on GCE node
W20260806 06:26:48.517937 30746 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:26:48.518401 30731 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 06:26:48.518646 30731 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 06:26:48.519790 30731 hybrid_clock.cc:648] HybridClock initialized: now 1785997608519779 us; error 24 us; skew 500 ppm
Aug 06 06:26:48 dist-test-slave-4tdj krb5kdc[30146](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997608, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.12.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 06:26:48.522228 30731 init.cc:377] Logged in from keytab as oryx/127.21.12.131@KRBTEST.COM (short username oryx)
I20260806 06:26:48.522675 30731 webserver.cc:460] Webserver started at http://127.21.12.131:43011/ using document root <none> and password file <none>
I20260806 06:26:48.522890 30731 fs_manager.cc:359] Metadata directory not provided
I20260806 06:26:48.522938 30731 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 06:26:48.523057 30731 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 06:26:48.524107 30731 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785997581049705-21554-0/minicluster-data/ts-2/data/instance:
uuid: "79d4ece6336b48e59c3e65c1d11c3ded"
format_stamp: "Formatted at 2026-08-06 06:26:48 on dist-test-slave-4tdj"
server_key: "82c0cde3ca11677e7935a3dee2fca78d"
server_key_iv: "63b32f56a89f84661cf91ad3c7298acd"
server_key_version: "encryptionkey@0"
I20260806 06:26:48.524433 30731 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785997581049705-21554-0/minicluster-data/ts-2/wal/instance:
uuid: "79d4ece6336b48e59c3e65c1d11c3ded"
format_stamp: "Formatted at 2026-08-06 06:26:48 on dist-test-slave-4tdj"
server_key: "82c0cde3ca11677e7935a3dee2fca78d"
server_key_iv: "63b32f56a89f84661cf91ad3c7298acd"
server_key_version: "encryptionkey@0"
I20260806 06:26:48.525674 30731 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260806 06:26:48.526284 30757 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 06:26:48.526460 30731 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260806 06:26:48.526572 30731 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785997581049705-21554-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785997581049705-21554-0/minicluster-data/ts-2/wal
uuid: "79d4ece6336b48e59c3e65c1d11c3ded"
format_stamp: "Formatted at 2026-08-06 06:26:48 on dist-test-slave-4tdj"
server_key: "82c0cde3ca11677e7935a3dee2fca78d"
server_key_iv: "63b32f56a89f84661cf91ad3c7298acd"
server_key_version: "encryptionkey@0"
I20260806 06:26:48.526634 30731 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785997581049705-21554-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785997581049705-21554-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785997581049705-21554-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 06:26:48.547065 30731 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 06:26:48.549111 30731 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 06:26:48.549258 30731 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 06:26:48.549448 30731 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 06:26:48.549706 30731 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 06:26:48.549737 30731 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 06:26:48.549759 30731 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 06:26:48.549770 30731 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 06:26:48.555527 30731 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.131:46131
I20260806 06:26:48.555596 30952 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.131:46131 every 8 connection(s)
I20260806 06:26:48.555855 30731 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785997581049705-21554-0/minicluster-data/ts-2/data/info.pb
Aug 06 06:26:48 dist-test-slave-4tdj krb5kdc[30146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997608, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.12.131@KRBTEST.COM for oryx/127.21.12.190@KRBTEST.COM
I20260806 06:26:48.564071 21554 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHhLhT/build/release/bin/kudu as pid 30731
I20260806 06:26:48.564177 21554 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785997581049705-21554-0/minicluster-data/ts-2/wal/instance
I20260806 06:26:48.564246 30277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:48.556579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.131:40385 (local address 127.21.12.190:41635)
0806 06:26:48.556638 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0806 06:26:48.556641 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 06:26:48.556859 (+ 218us) server_negotiation.cc:407] Connection header received
0806 06:26:48.557307 (+ 448us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:26:48.557308 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:26:48.557318 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:26:48.557364 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:26:48.558258 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:48.558785 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:48.559682 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:48.559867 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:48.559919 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:48.561960 (+ 2041us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:26:48.561962 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:26:48.561963 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 06:26:48.561969 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:26:48.563138 (+ 1169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:26:48.563474 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:26:48.563475 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:26:48.563475 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:26:48.563519 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:26:48.563692 (+ 173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:26:48.563692 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:26:48.563692 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:26:48.563781 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:26:48.563824 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0806 06:26:48.564095 (+ 271us) server_negotiation.cc:299] Negotiation successful
0806 06:26:48.564157 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260806 06:26:48.564373 21554 external_mini_cluster.cc:1428] Setting key a8eae7c9e03b4d54531f89f4c8d68da7
I20260806 06:26:48.564520 30955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:48.556587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:41635 (local address 127.21.12.131:40385)
0806 06:26:48.556744 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0806 06:26:48.556780 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0806 06:26:48.557231 (+ 451us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:26:48.557431 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:26:48.557435 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:26:48.558002 (+ 567us) client_negotiation.cc:189] Negotiated authn=SASL
0806 06:26:48.558179 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:48.558184 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:48.558852 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:48.558852 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:48.559611 (+ 759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:48.559613 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:48.559965 (+ 352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:48.559967 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:48.560089 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:48.560490 (+ 401us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 06:26:48.560498 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0806 06:26:48.561852 (+ 1354us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 06:26:48.563208 (+ 1356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:26:48.563214 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:26:48.563216 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:26:48.563417 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:26:48.563560 (+ 143us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:26:48.563561 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:26:48.563561 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:26:48.563652 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:26:48.563821 (+ 169us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 06:26:48.563824 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 06:26:48.563960 (+ 136us) client_negotiation.cc:769] Sending connection context
0806 06:26:48.564065 (+ 105us) client_negotiation.cc:240] Negotiation successful
0806 06:26:48.564222 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":88,"threads_started":1}
I20260806 06:26:48.564930 30953 heartbeater.cc:347] Connected to a master server at 127.21.12.190:41635
I20260806 06:26:48.565028 30953 heartbeater.cc:464] Registering TS with master...
I20260806 06:26:48.565227 30953 heartbeater.cc:511] Master 127.21.12.190:41635 requested a full tablet report, sending...
I20260806 06:26:48.565582 30217 ts_manager.cc:194] Registered new tserver with Master: 79d4ece6336b48e59c3e65c1d11c3ded (127.21.12.131:46131)
I20260806 06:26:48.565932 30217 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.21.12.131@KRBTEST.COM'} at 127.21.12.131:40385
I20260806 06:26:48.575963 21554 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.21.12.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.12.189@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.12.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785997581049705-21554-0/krb5kdc/oryx_127.21.12.189.keytab.
Entry for principal oryx/127.21.12.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785997581049705-21554-0/krb5kdc/oryx_127.21.12.189.keytab.
I20260806 06:26:48.600919 21554 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHhLhT/build/release/bin/kudu
/tmp/dist-test-taskFHhLhT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785997581049705-21554-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785997581049705-21554-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785997581049705-21554-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785997581049705-21554-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.12.189:35823
--webserver_interface=127.21.12.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785997581049705-21554-0/krb5kdc/oryx_127.21.12.189.keytab
--principal=oryx/127.21.12.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.148:39091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.12.190:41635,127.21.12.189:35823
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785997581049705-21554-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785997581049705-21554-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785997581049705-21554-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 06:26:48.680564 30962 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 06:26:48.680760 30962 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 06:26:48.680781 30962 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 06:26:48.681854 30962 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 06:26:48.681886 30962 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 06:26:48.681900 30962 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 06:26:48.681910 30962 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 06:26:48.681921 30962 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 06:26:48.683413 30962 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.148:39091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785997581049705-21554-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785997581049705-21554-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.21.12.190:41635,127.21.12.189:35823
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785997581049705-21554-0/krb5kdc/oryx_127.21.12.189.keytab
--principal=oryx/127.21.12.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.12.189:35823
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785997581049705-21554-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.21.12.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30962
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785997581049705-21554-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 05:11:02 UTC on ccycloud.quasar-hvzbbz.root.comops.site
build id 4569617
W20260806 06:26:48.683565 30962 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.21.12.190:41635,127.21.12.189:35823'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260806 06:26:48.683686 30962 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 06:26:48.683857 30962 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 06:26:48.688489 30976 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:26:48.688481 30971 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:26:48.688490 30973 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:26:48.688764 30962 server_base.cc:1034] running on GCE node
I20260806 06:26:48.688980 30962 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 06:26:48.689275 30962 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 06:26:48.690411 30962 hybrid_clock.cc:648] HybridClock initialized: now 1785997608690396 us; error 27 us; skew 500 ppm
Aug 06 06:26:48 dist-test-slave-4tdj krb5kdc[30146](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997608, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.12.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 06:26:48.692754 30962 init.cc:377] Logged in from keytab as oryx/127.21.12.189@KRBTEST.COM (short username oryx)
I20260806 06:26:48.693171 30962 webserver.cc:460] Webserver started at http://127.21.12.189:35419/ using document root <none> and password file <none>
I20260806 06:26:48.693363 30962 fs_manager.cc:359] Metadata directory not provided
I20260806 06:26:48.693405 30962 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 06:26:48.693511 30962 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 06:26:48.694594 30962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785997581049705-21554-0/minicluster-data/master-1/data/instance:
uuid: "124d842e9e484f8f85ddbc038dc51a31"
format_stamp: "Formatted at 2026-08-06 06:26:48 on dist-test-slave-4tdj"
server_key: "736fe95016f8c9697df2cc2400cd0ec6"
server_key_iv: "ab4a1aef45669ba8b5c0db557aa7011f"
server_key_version: "encryptionkey@0"
I20260806 06:26:48.694911 30962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785997581049705-21554-0/minicluster-data/master-1/wal/instance:
uuid: "124d842e9e484f8f85ddbc038dc51a31"
format_stamp: "Formatted at 2026-08-06 06:26:48 on dist-test-slave-4tdj"
server_key: "736fe95016f8c9697df2cc2400cd0ec6"
server_key_iv: "ab4a1aef45669ba8b5c0db557aa7011f"
server_key_version: "encryptionkey@0"
I20260806 06:26:48.696141 30962 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260806 06:26:48.696820 30989 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 06:26:48.697008 30962 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260806 06:26:48.697100 30962 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785997581049705-21554-0/minicluster-data/master-1/data,/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785997581049705-21554-0/minicluster-data/master-1/wal
uuid: "124d842e9e484f8f85ddbc038dc51a31"
format_stamp: "Formatted at 2026-08-06 06:26:48 on dist-test-slave-4tdj"
server_key: "736fe95016f8c9697df2cc2400cd0ec6"
server_key_iv: "ab4a1aef45669ba8b5c0db557aa7011f"
server_key_version: "encryptionkey@0"
I20260806 06:26:48.697151 30962 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785997581049705-21554-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785997581049705-21554-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785997581049705-21554-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 06:26:48.721246 30962 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 06:26:48.723467 30962 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 06:26:48.723624 30962 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 06:26:48.727795 30962 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.189:35823
I20260806 06:26:48.727835 31066 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.189:35823 every 8 connection(s)
I20260806 06:26:48.728176 30962 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785997581049705-21554-0/minicluster-data/master-1/data/info.pb
I20260806 06:26:48.729188 31067 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 06:26:48.730767 31067 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.12.190" port: 41635 } has no permanent_uuid. Determining permanent_uuid...
I20260806 06:26:48.735646 21554 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHhLhT/build/release/bin/kudu as pid 30962
Aug 06 06:26:48 dist-test-slave-4tdj krb5kdc[30146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997608, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.12.189@KRBTEST.COM for oryx/127.21.12.190@KRBTEST.COM
I20260806 06:26:48.739426 30277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:48.731425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50708 (local address 127.21.12.190:41635)
0806 06:26:48.731476 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0806 06:26:48.731478 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:26:48.731674 (+ 196us) server_negotiation.cc:407] Connection header received
0806 06:26:48.732145 (+ 471us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:26:48.732147 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:26:48.732161 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:26:48.732210 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:26:48.732816 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:48.733321 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:48.734328 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:48.734516 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:48.734573 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:48.736886 (+ 2313us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:26:48.736888 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:26:48.736888 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 06:26:48.736897 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:26:48.738164 (+ 1267us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:26:48.738560 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:26:48.738561 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:26:48.738561 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:26:48.738616 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:26:48.738825 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:26:48.738826 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:26:48.738826 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:26:48.738974 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:26:48.739049 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0806 06:26:48.739297 (+ 248us) server_negotiation.cc:299] Negotiation successful
0806 06:26:48.739361 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260806 06:26:48.739666 31071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:48.731390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:41635 (local address 127.0.0.1:50708)
0806 06:26:48.731557 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0806 06:26:48.731593 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0806 06:26:48.732046 (+ 453us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:26:48.732289 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:26:48.732293 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:26:48.732547 (+ 254us) client_negotiation.cc:189] Negotiated authn=SASL
0806 06:26:48.732731 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:48.732734 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:48.733403 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:48.733405 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:48.734225 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:48.734227 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:48.734584 (+ 357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:48.734586 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:48.734714 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:48.735137 (+ 423us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 06:26:48.735146 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0806 06:26:48.736785 (+ 1639us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 06:26:48.738237 (+ 1452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:26:48.738244 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:26:48.738246 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:26:48.738494 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:26:48.738668 (+ 174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:26:48.738669 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:26:48.738669 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:26:48.738771 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:26:48.739038 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 06:26:48.739043 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 06:26:48.739164 (+ 121us) client_negotiation.cc:769] Sending connection context
0806 06:26:48.739277 (+ 113us) client_negotiation.cc:240] Negotiation successful
0806 06:26:48.739427 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":76,"threads_started":1}
I20260806 06:26:48.740250 31067 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.12.189" port: 35823 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 06:26:48 dist-test-slave-4tdj krb5kdc[30146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997607, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.21.12.189@KRBTEST.COM
Aug 06 06:26:48 dist-test-slave-4tdj krb5kdc[30146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997608, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.12.189@KRBTEST.COM for oryx/127.21.12.189@KRBTEST.COM
I20260806 06:26:48.743907 31073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:48.736127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59408 (local address 127.21.12.189:35823)
0806 06:26:48.736409 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0806 06:26:48.736414 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 06:26:48.736437 (+ 23us) server_negotiation.cc:407] Connection header received
0806 06:26:48.736458 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:26:48.736460 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:26:48.736479 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:26:48.736557 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:26:48.737089 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:48.737738 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:48.738458 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:48.738752 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:48.738808 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:48.741208 (+ 2400us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:26:48.741216 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:26:48.741216 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 06:26:48.741223 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:26:48.742829 (+ 1606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:26:48.743175 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:26:48.743178 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:26:48.743179 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:26:48.743239 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:26:48.743397 (+ 158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:26:48.743398 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:26:48.743398 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:26:48.743529 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:26:48.743697 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0806 06:26:48.743740 (+ 43us) server_negotiation.cc:299] Negotiation successful
0806 06:26:48.743810 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":249,"thread_start_us":63,"threads_started":1}
I20260806 06:26:48.746589 31071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:48.740469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.189:35823 (local address 127.0.0.1:59420)
0806 06:26:48.740506 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0806 06:26:48.740516 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0806 06:26:48.740549 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:26:48.740742 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:26:48.740743 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:26:48.740876 (+ 133us) client_negotiation.cc:189] Negotiated authn=SASL
0806 06:26:48.740920 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:48.740922 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:48.741636 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:48.741637 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:48.742206 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:48.742208 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:48.742692 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:48.742693 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:48.742749 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:48.743064 (+ 315us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 06:26:48.743068 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0806 06:26:48.744120 (+ 1052us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 06:26:48.745873 (+ 1753us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:26:48.745873 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:26:48.745874 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:26:48.746019 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:26:48.746160 (+ 141us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:26:48.746160 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:26:48.746160 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:26:48.746201 (+ 41us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:26:48.746385 (+ 184us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 06:26:48.746386 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 06:26:48.746416 (+ 30us) client_negotiation.cc:769] Sending connection context
0806 06:26:48.746462 (+ 46us) client_negotiation.cc:240] Negotiation successful
0806 06:26:48.746527 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23}
I20260806 06:26:48.746688 31074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:48.740528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59420 (local address 127.21.12.189:35823)
0806 06:26:48.740660 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0806 06:26:48.740662 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:26:48.740666 (+ 4us) server_negotiation.cc:407] Connection header received
0806 06:26:48.740677 (+ 11us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:26:48.740677 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:26:48.740683 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:26:48.740731 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:26:48.740981 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:48.741557 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:48.742278 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:48.742614 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:48.742682 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:48.744195 (+ 1513us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:26:48.744197 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:26:48.744197 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 06:26:48.744203 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:26:48.745808 (+ 1605us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:26:48.746078 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:26:48.746078 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:26:48.746078 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:26:48.746120 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:26:48.746232 (+ 112us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:26:48.746233 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:26:48.746233 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:26:48.746333 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:26:48.746394 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0806 06:26:48.746477 (+ 83us) server_negotiation.cc:299] Negotiation successful
0806 06:26:48.746573 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":37,"server-negotiator.queue_time_us":119,"thread_start_us":98,"threads_started":1}
I20260806 06:26:48.747792 31067 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 124d842e9e484f8f85ddbc038dc51a31: Bootstrap starting.
I20260806 06:26:48.748363 31067 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 124d842e9e484f8f85ddbc038dc51a31: Neither blocks nor log segments found. Creating new log.
I20260806 06:26:48.748574 31067 log.cc:824] T 00000000000000000000000000000000 P 124d842e9e484f8f85ddbc038dc51a31: Log is configured to *not* fsync() on all Append() calls
I20260806 06:26:48.749224 31067 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 124d842e9e484f8f85ddbc038dc51a31: No bootstrap required, opened a new log
I20260806 06:26:48.750583 31067 raft_consensus.cc:348] T 00000000000000000000000000000000 P 124d842e9e484f8f85ddbc038dc51a31 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2589a7a56e404deda1060eb408ebf54c" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 41635 } } peers { permanent_uuid: "124d842e9e484f8f85ddbc038dc51a31" member_type: VOTER last_known_addr { host: "127.21.12.189" port: 35823 } }
I20260806 06:26:48.750698 31067 raft_consensus.cc:374] T 00000000000000000000000000000000 P 124d842e9e484f8f85ddbc038dc51a31 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 06:26:48.750718 31067 raft_consensus.cc:729] T 00000000000000000000000000000000 P 124d842e9e484f8f85ddbc038dc51a31 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 124d842e9e484f8f85ddbc038dc51a31, State: Initialized, Role: FOLLOWER
I20260806 06:26:48.750808 31067 consensus_queue.cc:260] T 00000000000000000000000000000000 P 124d842e9e484f8f85ddbc038dc51a31 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2589a7a56e404deda1060eb408ebf54c" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 41635 } } peers { permanent_uuid: "124d842e9e484f8f85ddbc038dc51a31" member_type: VOTER last_known_addr { host: "127.21.12.189" port: 35823 } }
I20260806 06:26:48.751014 31076 sys_catalog.cc:455] T 00000000000000000000000000000000 P 124d842e9e484f8f85ddbc038dc51a31 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2589a7a56e404deda1060eb408ebf54c" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 41635 } } peers { permanent_uuid: "124d842e9e484f8f85ddbc038dc51a31" member_type: VOTER last_known_addr { host: "127.21.12.189" port: 35823 } } }
I20260806 06:26:48.751075 31067 sys_catalog.cc:564] T 00000000000000000000000000000000 P 124d842e9e484f8f85ddbc038dc51a31 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 06:26:48.751098 31076 sys_catalog.cc:458] T 00000000000000000000000000000000 P 124d842e9e484f8f85ddbc038dc51a31 [sys.catalog]: This master's current role is: FOLLOWER
W20260806 06:26:48.752745 31094 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 124d842e9e484f8f85ddbc038dc51a31: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 06:26:48.752840 31094 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 06:26:48.754138 30962 master_runner.cc:428] Detected that this master 124d842e9e484f8f85ddbc038dc51a31 is joining an existing cluster
I20260806 06:26:48.754242 30962 master_runner.cc:432] Initiating AddMaster RPC to add 127.21.12.189:35823
I20260806 06:26:48.761297 31101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:48.755171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:41635 (local address 127.0.0.1:50724)
0806 06:26:48.755381 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0806 06:26:48.755392 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0806 06:26:48.755469 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:26:48.755603 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:26:48.755604 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:26:48.755847 (+ 243us) client_negotiation.cc:189] Negotiated authn=SASL
0806 06:26:48.755916 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:48.755918 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:48.756821 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:48.756822 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:48.757406 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:48.757408 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:48.757764 (+ 356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:48.757765 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:48.757825 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:48.758212 (+ 387us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 06:26:48.758217 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0806 06:26:48.758624 (+ 407us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 06:26:48.760277 (+ 1653us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:26:48.760278 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:26:48.760279 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:26:48.760503 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:26:48.760674 (+ 171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:26:48.760675 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:26:48.760675 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:26:48.760761 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:26:48.760973 (+ 212us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 06:26:48.760974 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 06:26:48.761015 (+ 41us) client_negotiation.cc:769] Sending connection context
0806 06:26:48.761089 (+ 74us) client_negotiation.cc:240] Negotiation successful
0806 06:26:48.761187 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":188,"mutex_wait_us":32,"thread_start_us":151,"threads_started":1}
I20260806 06:26:48.761312 30277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:48.755210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50724 (local address 127.21.12.190:41635)
0806 06:26:48.755248 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0806 06:26:48.755250 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:26:48.755447 (+ 197us) server_negotiation.cc:407] Connection header received
0806 06:26:48.755542 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:26:48.755543 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:26:48.755554 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:26:48.755598 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:26:48.755983 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:48.756745 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:48.757483 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:48.757684 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:48.757736 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:48.758710 (+ 974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:26:48.758711 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:26:48.758712 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 06:26:48.758718 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:26:48.760205 (+ 1487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:26:48.760567 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:26:48.760568 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:26:48.760569 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:26:48.760623 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:26:48.760811 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:26:48.760812 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:26:48.760812 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:26:48.760919 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:26:48.760971 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0806 06:26:48.761135 (+ 164us) server_negotiation.cc:299] Negotiation successful
0806 06:26:48.761223 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260806 06:26:48.761701 31074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:48.755218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59434 (local address 127.21.12.189:35823)
0806 06:26:48.755251 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0806 06:26:48.755253 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:26:48.755456 (+ 203us) server_negotiation.cc:407] Connection header received
0806 06:26:48.755501 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:26:48.755501 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:26:48.755511 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:26:48.755564 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:26:48.756013 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:48.756733 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:48.757772 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:48.758061 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:48.758102 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:48.759070 (+ 968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:26:48.759071 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:26:48.759072 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 06:26:48.759079 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:26:48.760628 (+ 1549us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:26:48.761023 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:26:48.761024 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:26:48.761024 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:26:48.761076 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:26:48.761245 (+ 169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:26:48.761246 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:26:48.761247 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:26:48.761348 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:26:48.761389 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0806 06:26:48.761559 (+ 170us) server_negotiation.cc:299] Negotiation successful
0806 06:26:48.761630 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260806 06:26:48.761898 31102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:48.755281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.189:35823 (local address 127.0.0.1:59434)
0806 06:26:48.755392 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0806 06:26:48.755400 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0806 06:26:48.755452 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:26:48.755598 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:26:48.755600 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:26:48.755865 (+ 265us) client_negotiation.cc:189] Negotiated authn=SASL
0806 06:26:48.755926 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:48.755927 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:48.756823 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:48.756824 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:48.757677 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:48.757679 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:48.758165 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:48.758166 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:48.758219 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:48.758617 (+ 398us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 06:26:48.758623 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0806 06:26:48.758968 (+ 345us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 06:26:48.760711 (+ 1743us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:26:48.760711 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:26:48.760712 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:26:48.760936 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:26:48.761139 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:26:48.761140 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:26:48.761140 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:26:48.761188 (+ 48us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:26:48.761427 (+ 239us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 06:26:48.761428 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 06:26:48.761460 (+ 32us) client_negotiation.cc:769] Sending connection context
0806 06:26:48.761761 (+ 301us) client_negotiation.cc:240] Negotiation successful
0806 06:26:48.761839 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":75,"threads_started":1}
Aug 06 06:26:48 dist-test-slave-4tdj krb5kdc[30146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997608, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.12.190@KRBTEST.COM for oryx/127.21.12.189@KRBTEST.COM
I20260806 06:26:48.770117 31103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:48.763673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.189:35823 (local address 127.0.0.1:59446)
0806 06:26:48.763794 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0806 06:26:48.763834 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0806 06:26:48.763893 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:26:48.764026 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:26:48.764028 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:26:48.764264 (+ 236us) client_negotiation.cc:189] Negotiated authn=SASL
0806 06:26:48.764308 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:48.764309 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:48.765132 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:48.765133 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:48.765711 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:48.765713 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:48.766123 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:48.766123 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:48.766176 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:48.766588 (+ 412us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 06:26:48.766592 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0806 06:26:48.767766 (+ 1174us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 06:26:48.769213 (+ 1447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:26:48.769220 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:26:48.769221 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:26:48.769398 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:26:48.769573 (+ 175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:26:48.769573 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:26:48.769573 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:26:48.769622 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:26:48.769832 (+ 210us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 06:26:48.769837 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 06:26:48.769874 (+ 37us) client_negotiation.cc:769] Sending connection context
0806 06:26:48.769931 (+ 57us) client_negotiation.cc:240] Negotiation successful
0806 06:26:48.770014 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":62,"threads_started":1}
I20260806 06:26:48.770129 31074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:48.763748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59446 (local address 127.21.12.189:35823)
0806 06:26:48.763780 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0806 06:26:48.763782 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:26:48.763888 (+ 106us) server_negotiation.cc:407] Connection header received
0806 06:26:48.763958 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:26:48.763959 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:26:48.763968 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:26:48.764024 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:26:48.764367 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:48.765063 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:48.765779 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:48.766062 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:48.766125 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:48.767839 (+ 1714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:26:48.767841 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:26:48.767841 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 06:26:48.767847 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:26:48.769142 (+ 1295us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:26:48.769458 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:26:48.769459 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:26:48.769460 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:26:48.769512 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:26:48.769668 (+ 156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:26:48.769669 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:26:48.769669 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:26:48.769782 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:26:48.769843 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0806 06:26:48.769975 (+ 132us) server_negotiation.cc:299] Negotiation successful
0806 06:26:48.770043 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260806 06:26:48.770634 30217 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.21.12.189:35823: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "124d842e9e484f8f85ddbc038dc51a31"
member_type: NON_VOTER
last_known_addr {
host: "127.21.12.189"
port: 35823
}
attrs {
promote: true
}
}
dest_uuid: "2589a7a56e404deda1060eb408ebf54c"
cas_config_opid_index: -1
I20260806 06:26:48.770949 30217 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2589a7a56e404deda1060eb408ebf54c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "2589a7a56e404deda1060eb408ebf54c" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 41635 } } peers { permanent_uuid: "124d842e9e484f8f85ddbc038dc51a31" member_type: NON_VOTER last_known_addr { host: "127.21.12.189" port: 35823 } attrs { promote: true } }
I20260806 06:26:48.771428 31107 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 2589a7a56e404deda1060eb408ebf54c [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 124d842e9e484f8f85ddbc038dc51a31 (127.21.12.189) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "2589a7a56e404deda1060eb408ebf54c" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 41635 } } peers { permanent_uuid: "124d842e9e484f8f85ddbc038dc51a31" member_type: NON_VOTER last_known_addr { host: "127.21.12.189" port: 35823 } attrs { promote: true } } }
I20260806 06:26:48.771668 31028 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 124d842e9e484f8f85ddbc038dc51a31 [term 0 FOLLOWER]: Advancing to term 1
I20260806 06:26:48.772397 31028 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 124d842e9e484f8f85ddbc038dc51a31 [term 1 FOLLOWER]: Refusing update from remote peer 2589a7a56e404deda1060eb408ebf54c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260806 06:26:48.772416 31107 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.21.12.189:35823
I20260806 06:26:48.772490 31107 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2589a7a56e404deda1060eb408ebf54c [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "2589a7a56e404deda1060eb408ebf54c" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "2589a7a56e404deda1060eb408ebf54c" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 41635 } } peers { permanent_uuid: "124d842e9e484f8f85ddbc038dc51a31" member_type: NON_VOTER last_known_addr { host: "127.21.12.189" port: 35823 } attrs { promote: true } } }
I20260806 06:26:48.772564 31107 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2589a7a56e404deda1060eb408ebf54c [sys.catalog]: This master's current role is: LEADER
I20260806 06:26:48.772792 30962 master.cc:537] Master@127.21.12.189:35823 shutting down...
I20260806 06:26:48.773327 31106 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2589a7a56e404deda1060eb408ebf54c [LEADER]: Connected to new peer: Peer: permanent_uuid: "124d842e9e484f8f85ddbc038dc51a31" member_type: NON_VOTER last_known_addr { host: "127.21.12.189" port: 35823 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260806 06:26:48.774129 30189 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260806 06:26:48.774580 30189 consensus_peers.cc:487] T 00000000000000000000000000000000 P 2589a7a56e404deda1060eb408ebf54c -> Peer 124d842e9e484f8f85ddbc038dc51a31 (127.21.12.189:35823): Couldn't send request to peer 124d842e9e484f8f85ddbc038dc51a31. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260806 06:26:48.775346 30962 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 124d842e9e484f8f85ddbc038dc51a31 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 06:26:48.775424 30962 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 124d842e9e484f8f85ddbc038dc51a31 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 06:26:48.775473 30962 tablet_replica.cc:330] T 00000000000000000000000000000000 P 124d842e9e484f8f85ddbc038dc51a31: stopping tablet replica
I20260806 06:26:48.787846 30962 master.cc:552] Master@127.21.12.189:35823 shutdown complete.
I20260806 06:26:48.787889 30962 master_runner.cc:305] Clearing existing system tablet
I20260806 06:26:48.788002 30962 fs_manager.cc:359] Metadata directory not provided
I20260806 06:26:48.788043 30962 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 06:26:48.788573 30962 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260806 06:26:48.788987 31115 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 06:26:48.789129 30962 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785997581049705-21554-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 06:26:48.789183 30962 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260806 06:26:48.789206 30962 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785997581049705-21554-0/minicluster-data/master-1/data,/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785997581049705-21554-0/minicluster-data/master-1/wal
uuid: "124d842e9e484f8f85ddbc038dc51a31"
format_stamp: "Formatted at 2026-08-06 06:26:48 on dist-test-slave-4tdj"
server_key: "736fe95016f8c9697df2cc2400cd0ec6"
server_key_iv: "ab4a1aef45669ba8b5c0db557aa7011f"
server_key_version: "encryptionkey@0"
I20260806 06:26:48.789866 30962 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 124d842e9e484f8f85ddbc038dc51a31: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 06:26:48.791679 30962 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 124d842e9e484f8f85ddbc038dc51a31: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260806 06:26:48.791749 30962 log.cc:1196] T 00000000000000000000000000000000 P 124d842e9e484f8f85ddbc038dc51a31: Deleting WAL directory at /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785997581049705-21554-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260806 06:26:48.791949 30962 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 124d842e9e484f8f85ddbc038dc51a31: Deleting consensus metadata
I20260806 06:26:48.792059 30962 master_runner.cc:315] Copying system tablet from 127.21.12.190:41635
I20260806 06:26:48.792618 30962 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 124d842e9e484f8f85ddbc038dc51a31: tablet copy: Beginning tablet copy session from remote peer at address 127.21.12.190:41635
I20260806 06:26:48.798739 31124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:48.792892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:41635 (local address 127.0.0.1:50740)
0806 06:26:48.792997 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0806 06:26:48.793008 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0806 06:26:48.793067 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:26:48.793200 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:26:48.793200 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:26:48.793362 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0806 06:26:48.793415 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:48.793416 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:48.794360 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:48.794361 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:48.795056 (+ 695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:48.795059 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:48.795505 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:48.795505 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:48.795550 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:48.795958 (+ 408us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 06:26:48.795963 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0806 06:26:48.796260 (+ 297us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 06:26:48.797827 (+ 1567us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:26:48.797828 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:26:48.797828 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:26:48.797986 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:26:48.798164 (+ 178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:26:48.798164 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:26:48.798164 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:26:48.798213 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:26:48.798466 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 06:26:48.798466 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 06:26:48.798503 (+ 37us) client_negotiation.cc:769] Sending connection context
0806 06:26:48.798568 (+ 65us) client_negotiation.cc:240] Negotiation successful
0806 06:26:48.798670 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":82,"thread_start_us":65,"threads_started":1}
I20260806 06:26:48.798774 30277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:48.792963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50740 (local address 127.21.12.190:41635)
0806 06:26:48.793011 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0806 06:26:48.793013 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:26:48.793066 (+ 53us) server_negotiation.cc:407] Connection header received
0806 06:26:48.793112 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:26:48.793113 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:26:48.793124 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:26:48.793195 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:26:48.793471 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:48.794241 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:48.795131 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:48.795439 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:48.795511 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:48.796333 (+ 822us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:26:48.796335 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:26:48.796335 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 06:26:48.796341 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:26:48.797745 (+ 1404us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:26:48.798044 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:26:48.798045 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:26:48.798046 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:26:48.798106 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:26:48.798262 (+ 156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:26:48.798262 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:26:48.798263 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:26:48.798408 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:26:48.798477 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0806 06:26:48.798601 (+ 124us) server_negotiation.cc:299] Negotiation successful
0806 06:26:48.798680 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260806 06:26:48.799188 30250 tablet_copy_service.cc:140] P 2589a7a56e404deda1060eb408ebf54c: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 124d842e9e484f8f85ddbc038dc51a31 ({username='oryx', principal='oryx/127.21.12.189@KRBTEST.COM'} at 127.0.0.1:50740)
I20260806 06:26:48.799273 30250 tablet_copy_service.cc:161] P 2589a7a56e404deda1060eb408ebf54c: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 124d842e9e484f8f85ddbc038dc51a31 at {username='oryx', principal='oryx/127.21.12.189@KRBTEST.COM'} at 127.0.0.1:50740: session id = 124d842e9e484f8f85ddbc038dc51a31-00000000000000000000000000000000
I20260806 06:26:48.799893 30250 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 2589a7a56e404deda1060eb408ebf54c: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 06:26:48.800446 30962 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260806 06:26:48.801265 30962 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 124d842e9e484f8f85ddbc038dc51a31: tablet copy: Starting download of 0 data blocks...
I20260806 06:26:48.801386 30962 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 124d842e9e484f8f85ddbc038dc51a31: tablet copy: Starting download of 1 WAL segments...
I20260806 06:26:48.802174 30962 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 124d842e9e484f8f85ddbc038dc51a31: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 06:26:48.802901 30250 tablet_copy_service.cc:342] P 2589a7a56e404deda1060eb408ebf54c: Request end of tablet copy session 124d842e9e484f8f85ddbc038dc51a31-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.21.12.189@KRBTEST.COM'} at 127.0.0.1:50740
I20260806 06:26:48.802953 30250 tablet_copy_service.cc:433] P 2589a7a56e404deda1060eb408ebf54c: ending tablet copy session 124d842e9e484f8f85ddbc038dc51a31-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 124d842e9e484f8f85ddbc038dc51a31
I20260806 06:26:48.805022 30962 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 06:26:48.806108 31128 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:26:48.806202 31132 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:26:48.806108 31125 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:26:48.806416 30962 server_base.cc:1034] running on GCE node
I20260806 06:26:48.806493 30962 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 06:26:48.806609 30962 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 06:26:48.807719 30962 hybrid_clock.cc:648] HybridClock initialized: now 1785997608807708 us; error 25 us; skew 500 ppm
Aug 06 06:26:48 dist-test-slave-4tdj krb5kdc[30146](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997608, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.12.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 06:26:48.808938 30962 init.cc:377] Logged in from keytab as oryx/127.21.12.189@KRBTEST.COM (short username oryx)
I20260806 06:26:48.809188 30962 webserver.cc:460] Webserver started at http://127.21.12.189:43891/ using document root <none> and password file <none>
I20260806 06:26:48.809301 30962 fs_manager.cc:359] Metadata directory not provided
I20260806 06:26:48.809335 30962 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 06:26:48.809962 30962 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.000s
I20260806 06:26:48.810487 31143 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 06:26:48.810592 30962 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260806 06:26:48.810634 30962 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785997581049705-21554-0/minicluster-data/master-1/data,/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785997581049705-21554-0/minicluster-data/master-1/wal
uuid: "124d842e9e484f8f85ddbc038dc51a31"
format_stamp: "Formatted at 2026-08-06 06:26:48 on dist-test-slave-4tdj"
server_key: "736fe95016f8c9697df2cc2400cd0ec6"
server_key_iv: "ab4a1aef45669ba8b5c0db557aa7011f"
server_key_version: "encryptionkey@0"
I20260806 06:26:48.810683 30962 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785997581049705-21554-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785997581049705-21554-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785997581049705-21554-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 06:26:48.825812 30962 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 06:26:48.827354 30962 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 06:26:48.829826 30962 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.189:35823
I20260806 06:26:48.830101 31217 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.189:35823 every 8 connection(s)
I20260806 06:26:48.830210 30962 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785997581049705-21554-0/minicluster-data/master-1/data/info.pb
I20260806 06:26:48.831005 31218 sys_catalog.cc:263] Verifying existing consensus state
I20260806 06:26:48.831419 31218 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 124d842e9e484f8f85ddbc038dc51a31: Bootstrap starting.
I20260806 06:26:48.833509 31218 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 124d842e9e484f8f85ddbc038dc51a31: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 06:26:48.833714 31218 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 124d842e9e484f8f85ddbc038dc51a31: Bootstrap complete.
I20260806 06:26:48.833904 31218 raft_consensus.cc:348] T 00000000000000000000000000000000 P 124d842e9e484f8f85ddbc038dc51a31 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "2589a7a56e404deda1060eb408ebf54c" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 41635 } } peers { permanent_uuid: "124d842e9e484f8f85ddbc038dc51a31" member_type: NON_VOTER last_known_addr { host: "127.21.12.189" port: 35823 } attrs { promote: true } }
I20260806 06:26:48.833971 31218 raft_consensus.cc:729] T 00000000000000000000000000000000 P 124d842e9e484f8f85ddbc038dc51a31 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 124d842e9e484f8f85ddbc038dc51a31, State: Initialized, Role: LEARNER
I20260806 06:26:48.833997 31218 consensus_queue.cc:260] T 00000000000000000000000000000000 P 124d842e9e484f8f85ddbc038dc51a31 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "2589a7a56e404deda1060eb408ebf54c" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 41635 } } peers { permanent_uuid: "124d842e9e484f8f85ddbc038dc51a31" member_type: NON_VOTER last_known_addr { host: "127.21.12.189" port: 35823 } attrs { promote: true } }
I20260806 06:26:48.834144 31218 sys_catalog.cc:564] T 00000000000000000000000000000000 P 124d842e9e484f8f85ddbc038dc51a31 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 06:26:48.834167 31223 sys_catalog.cc:455] T 00000000000000000000000000000000 P 124d842e9e484f8f85ddbc038dc51a31 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "2589a7a56e404deda1060eb408ebf54c" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 41635 } } peers { permanent_uuid: "124d842e9e484f8f85ddbc038dc51a31" member_type: NON_VOTER last_known_addr { host: "127.21.12.189" port: 35823 } attrs { promote: true } } }
I20260806 06:26:48.834270 31223 sys_catalog.cc:458] T 00000000000000000000000000000000 P 124d842e9e484f8f85ddbc038dc51a31 [sys.catalog]: This master's current role is: LEARNER
I20260806 06:26:48.835464 31238 catalog_manager.cc:1259] Loaded cluster ID: 2793aaf3eaab49c4806dfa641e5153fa
I20260806 06:26:48.835507 31238 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 124d842e9e484f8f85ddbc038dc51a31: loading cluster ID for follower catalog manager: success
I20260806 06:26:48.836480 31238 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 124d842e9e484f8f85ddbc038dc51a31: acquiring CA information for follower catalog manager: success
I20260806 06:26:48.836728 31238 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 124d842e9e484f8f85ddbc038dc51a31: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260806 06:26:48.851132 31239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:48.845381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59468 (local address 127.21.12.189:35823)
0806 06:26:48.845496 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0806 06:26:48.845498 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:26:48.845505 (+ 7us) server_negotiation.cc:407] Connection header received
0806 06:26:48.845525 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:26:48.845525 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:26:48.845534 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:26:48.845586 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:26:48.846075 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:48.846795 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:48.847475 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:48.847679 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:48.847734 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:48.848829 (+ 1095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:26:48.848831 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:26:48.848831 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 06:26:48.848839 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:26:48.850062 (+ 1223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:26:48.850403 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:26:48.850404 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:26:48.850404 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:26:48.850457 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:26:48.850621 (+ 164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:26:48.850622 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:26:48.850622 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:26:48.850738 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:26:48.850824 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0806 06:26:48.850932 (+ 108us) server_negotiation.cc:299] Negotiation successful
0806 06:26:48.851011 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97,"thread_start_us":61,"threads_started":1}
WARNING: no policy specified for oryx/127.21.12.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.12.188@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.12.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785997581049705-21554-0/krb5kdc/oryx_127.21.12.188.keytab.
Entry for principal oryx/127.21.12.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785997581049705-21554-0/krb5kdc/oryx_127.21.12.188.keytab.
I20260806 06:26:48.877331 21554 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHhLhT/build/release/bin/kudu
/tmp/dist-test-taskFHhLhT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785997581049705-21554-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785997581049705-21554-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785997581049705-21554-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785997581049705-21554-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.12.188:35289
--webserver_interface=127.21.12.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785997581049705-21554-0/krb5kdc/oryx_127.21.12.188.keytab
--principal=oryx/127.21.12.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.148:39091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.12.190:41635,127.21.12.189:35823,127.21.12.188:35289
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785997581049705-21554-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785997581049705-21554-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785997581049705-21554-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 06:26:48.976681 31242 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 06:26:48.976874 31242 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 06:26:48.976895 31242 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 06:26:48.978030 31242 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 06:26:48.978080 31242 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 06:26:48.978103 31242 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 06:26:48.978135 31242 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 06:26:48.978156 31242 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 06:26:48.979808 31242 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.148:39091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785997581049705-21554-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785997581049705-21554-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.21.12.190:41635,127.21.12.189:35823,127.21.12.188:35289
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785997581049705-21554-0/krb5kdc/oryx_127.21.12.188.keytab
--principal=oryx/127.21.12.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.12.188:35289
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785997581049705-21554-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.21.12.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31242
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785997581049705-21554-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 05:11:02 UTC on ccycloud.quasar-hvzbbz.root.comops.site
build id 4569617
I20260806 06:26:48.980046 31242 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 06:26:48.980255 31242 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 06:26:48.984898 31254 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:26:48.984911 31249 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:26:48.985054 31242 server_base.cc:1034] running on GCE node
W20260806 06:26:48.984911 31252 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:26:48.985325 31242 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 06:26:48.985581 31242 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 06:26:48.986709 31242 hybrid_clock.cc:648] HybridClock initialized: now 1785997608986693 us; error 32 us; skew 500 ppm
Aug 06 06:26:48 dist-test-slave-4tdj krb5kdc[30146](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997608, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.12.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 06:26:48.992197 31242 init.cc:377] Logged in from keytab as oryx/127.21.12.188@KRBTEST.COM (short username oryx)
I20260806 06:26:48.992630 31242 webserver.cc:460] Webserver started at http://127.21.12.188:40329/ using document root <none> and password file <none>
I20260806 06:26:48.992830 31242 fs_manager.cc:359] Metadata directory not provided
I20260806 06:26:48.992875 31242 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 06:26:48.992997 31242 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 06:26:48.994021 31242 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785997581049705-21554-0/minicluster-data/master-2/data/instance:
uuid: "c88b3807d6bd413bb876555a253b8680"
format_stamp: "Formatted at 2026-08-06 06:26:48 on dist-test-slave-4tdj"
server_key: "0147de3e322a27788c3f07ea29117340"
server_key_iv: "523e6807d20dabc92705fbdee875d122"
server_key_version: "encryptionkey@0"
I20260806 06:26:48.994403 31242 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785997581049705-21554-0/minicluster-data/master-2/wal/instance:
uuid: "c88b3807d6bd413bb876555a253b8680"
format_stamp: "Formatted at 2026-08-06 06:26:48 on dist-test-slave-4tdj"
server_key: "0147de3e322a27788c3f07ea29117340"
server_key_iv: "523e6807d20dabc92705fbdee875d122"
server_key_version: "encryptionkey@0"
I20260806 06:26:48.995723 31242 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260806 06:26:48.996397 31266 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 06:26:48.996541 31242 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260806 06:26:48.996623 31242 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785997581049705-21554-0/minicluster-data/master-2/data,/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785997581049705-21554-0/minicluster-data/master-2/wal
uuid: "c88b3807d6bd413bb876555a253b8680"
format_stamp: "Formatted at 2026-08-06 06:26:48 on dist-test-slave-4tdj"
server_key: "0147de3e322a27788c3f07ea29117340"
server_key_iv: "523e6807d20dabc92705fbdee875d122"
server_key_version: "encryptionkey@0"
I20260806 06:26:48.996685 31242 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785997581049705-21554-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785997581049705-21554-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785997581049705-21554-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 06:26:49.017302 31242 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 06:26:49.019495 31242 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 06:26:49.019690 31242 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 06:26:49.030151 31242 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.188:35289
I20260806 06:26:49.030336 31353 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.188:35289 every 8 connection(s)
I20260806 06:26:49.030624 31242 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785997581049705-21554-0/minicluster-data/master-2/data/info.pb
I20260806 06:26:49.031735 31354 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 06:26:49.033028 21554 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHhLhT/build/release/bin/kudu as pid 31242
I20260806 06:26:49.033958 31354 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.12.190" port: 41635 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 06:26:49 dist-test-slave-4tdj krb5kdc[30146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997608, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.12.188@KRBTEST.COM for oryx/127.21.12.190@KRBTEST.COM
Aug 06 06:26:49 dist-test-slave-4tdj krb5kdc[30146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997607, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.21.12.188@KRBTEST.COM
I20260806 06:26:49.043524 30277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:49.035244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50756 (local address 127.21.12.190:41635)
0806 06:26:49.035287 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0806 06:26:49.035289 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:26:49.035449 (+ 160us) server_negotiation.cc:407] Connection header received
0806 06:26:49.035554 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:26:49.035555 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:26:49.035567 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:26:49.035615 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:26:49.036264 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:49.036930 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:49.038054 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:49.038356 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:49.038413 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:49.040674 (+ 2261us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:26:49.040676 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:26:49.040676 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 06:26:49.040685 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:26:49.042165 (+ 1480us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:26:49.042620 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:26:49.042620 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:26:49.042621 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:26:49.042669 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:26:49.042905 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:26:49.042905 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:26:49.042906 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:26:49.043027 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:26:49.043095 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0806 06:26:49.043354 (+ 259us) server_negotiation.cc:299] Negotiation successful
0806 06:26:49.043439 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260806 06:26:49.043748 31358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:49.035143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:41635 (local address 127.0.0.1:50756)
0806 06:26:49.035353 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0806 06:26:49.035389 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0806 06:26:49.035476 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:26:49.035672 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:26:49.035675 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:26:49.036017 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0806 06:26:49.036174 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:49.036179 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:49.037108 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:49.037109 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:49.037983 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:49.037986 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:49.038562 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:49.038564 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:49.038659 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:49.039056 (+ 397us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 06:26:49.039066 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0806 06:26:49.040589 (+ 1523us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 06:26:49.042239 (+ 1650us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:26:49.042244 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:26:49.042247 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:26:49.042549 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:26:49.042721 (+ 172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:26:49.042722 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:26:49.042723 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:26:49.042845 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:26:49.043086 (+ 241us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 06:26:49.043090 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 06:26:49.043212 (+ 122us) client_negotiation.cc:769] Sending connection context
0806 06:26:49.043325 (+ 113us) client_negotiation.cc:240] Negotiation successful
0806 06:26:49.043466 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":152,"threads_started":1}
I20260806 06:26:49.044227 31354 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.12.189" port: 35823 } has no permanent_uuid. Determining permanent_uuid...
I20260806 06:26:49.044342 31356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:49.033640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35290 (local address 127.21.12.188:35289)
0806 06:26:49.033834 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0806 06:26:49.033845 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 06:26:49.033874 (+ 29us) server_negotiation.cc:407] Connection header received
0806 06:26:49.034594 (+ 720us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:26:49.034604 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:26:49.034680 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:26:49.034824 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:26:49.035657 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:49.036721 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:49.037716 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:49.038114 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:49.038215 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:49.041116 (+ 2901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:26:49.041123 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:26:49.041123 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 06:26:49.041130 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:26:49.042856 (+ 1726us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:26:49.043206 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:26:49.043209 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:26:49.043209 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:26:49.043264 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:26:49.043448 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:26:49.043449 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:26:49.043449 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:26:49.043599 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:26:49.043720 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0806 06:26:49.044121 (+ 401us) server_negotiation.cc:299] Negotiation successful
0806 06:26:49.044192 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":108,"threads_started":1}
Aug 06 06:26:49 dist-test-slave-4tdj krb5kdc[30146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997608, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.12.188@KRBTEST.COM for oryx/127.21.12.189@KRBTEST.COM
I20260806 06:26:49.050487 31358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:49.044454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.189:35823 (local address 127.0.0.1:59470)
0806 06:26:49.044484 (+ 30us) negotiation.cc:107] Waiting for socket to connect
0806 06:26:49.044494 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0806 06:26:49.044529 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:26:49.044865 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:26:49.044866 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:26:49.044991 (+ 125us) client_negotiation.cc:189] Negotiated authn=SASL
0806 06:26:49.045073 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:49.045075 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:49.045692 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:49.045693 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:49.046182 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:49.046183 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:49.046527 (+ 344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:49.046528 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:49.046620 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:49.046921 (+ 301us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 06:26:49.046924 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0806 06:26:49.047971 (+ 1047us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 06:26:49.049584 (+ 1613us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:26:49.049585 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:26:49.049586 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:26:49.049746 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:26:49.049921 (+ 175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:26:49.049922 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:26:49.049922 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:26:49.049976 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:26:49.050187 (+ 211us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 06:26:49.050188 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 06:26:49.050227 (+ 39us) client_negotiation.cc:769] Sending connection context
0806 06:26:49.050287 (+ 60us) client_negotiation.cc:240] Negotiation successful
0806 06:26:49.050398 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15}
I20260806 06:26:49.050505 31239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:49.044702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59470 (local address 127.21.12.189:35823)
0806 06:26:49.044758 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0806 06:26:49.044760 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:26:49.044766 (+ 6us) server_negotiation.cc:407] Connection header received
0806 06:26:49.044787 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:26:49.044788 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:26:49.044799 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:26:49.044840 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:26:49.045142 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:49.045612 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:49.046236 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:49.046469 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:49.046512 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:49.048046 (+ 1534us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:26:49.048048 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:26:49.048048 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 06:26:49.048055 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:26:49.049376 (+ 1321us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:26:49.049813 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:26:49.049815 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:26:49.049815 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:26:49.049867 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:26:49.050025 (+ 158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:26:49.050026 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:26:49.050026 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:26:49.050133 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:26:49.050186 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0806 06:26:49.050335 (+ 149us) server_negotiation.cc:299] Negotiation successful
0806 06:26:49.050410 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260806 06:26:49.050894 31354 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.12.188" port: 35289 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 06:26:49 dist-test-slave-4tdj krb5kdc[30146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997608, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.12.188@KRBTEST.COM for oryx/127.21.12.188@KRBTEST.COM
I20260806 06:26:49.056304 31358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:49.051113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.188:35289 (local address 127.0.0.1:35302)
0806 06:26:49.051151 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0806 06:26:49.051161 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0806 06:26:49.051201 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:26:49.051298 (+ 97us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:26:49.051299 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:26:49.051427 (+ 128us) client_negotiation.cc:189] Negotiated authn=SASL
0806 06:26:49.051451 (+ 24us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:49.051452 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:49.052055 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:49.052056 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:49.052569 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:49.052571 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:49.052887 (+ 316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:49.052888 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:49.052921 (+ 33us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:49.053187 (+ 266us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 06:26:49.053190 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0806 06:26:49.054163 (+ 973us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 06:26:49.055412 (+ 1249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:26:49.055413 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:26:49.055413 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:26:49.055603 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:26:49.055831 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:26:49.055832 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:26:49.055832 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:26:49.055889 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:26:49.056074 (+ 185us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 06:26:49.056075 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 06:26:49.056105 (+ 30us) client_negotiation.cc:769] Sending connection context
0806 06:26:49.056168 (+ 63us) client_negotiation.cc:240] Negotiation successful
0806 06:26:49.056248 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24}
I20260806 06:26:49.056391 31356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:49.051141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35302 (local address 127.21.12.188:35289)
0806 06:26:49.051170 (+ 29us) server_negotiation.cc:206] Beginning negotiation
0806 06:26:49.051171 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0806 06:26:49.051189 (+ 18us) server_negotiation.cc:407] Connection header received
0806 06:26:49.051246 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:26:49.051247 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:26:49.051254 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:26:49.051295 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:26:49.051497 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:49.051981 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:49.052627 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:49.052832 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:49.052884 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:49.054229 (+ 1345us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:26:49.054230 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:26:49.054230 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 06:26:49.054236 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:26:49.055330 (+ 1094us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:26:49.055688 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:26:49.055689 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:26:49.055689 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:26:49.055746 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:26:49.055938 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:26:49.055939 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:26:49.055939 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:26:49.056030 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:26:49.056076 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0806 06:26:49.056185 (+ 109us) server_negotiation.cc:299] Negotiation successful
0806 06:26:49.056294 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":50,"server-negotiator.queue_time_us":13}
I20260806 06:26:49.057792 31354 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c88b3807d6bd413bb876555a253b8680: Bootstrap starting.
I20260806 06:26:49.059233 31354 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c88b3807d6bd413bb876555a253b8680: Neither blocks nor log segments found. Creating new log.
I20260806 06:26:49.059716 31354 log.cc:824] T 00000000000000000000000000000000 P c88b3807d6bd413bb876555a253b8680: Log is configured to *not* fsync() on all Append() calls
I20260806 06:26:49.060417 31354 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c88b3807d6bd413bb876555a253b8680: No bootstrap required, opened a new log
I20260806 06:26:49.062824 31354 raft_consensus.cc:348] T 00000000000000000000000000000000 P c88b3807d6bd413bb876555a253b8680 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2589a7a56e404deda1060eb408ebf54c" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 41635 } } peers { permanent_uuid: "124d842e9e484f8f85ddbc038dc51a31" member_type: VOTER last_known_addr { host: "127.21.12.189" port: 35823 } } peers { permanent_uuid: "c88b3807d6bd413bb876555a253b8680" member_type: VOTER last_known_addr { host: "127.21.12.188" port: 35289 } }
I20260806 06:26:49.062942 31354 raft_consensus.cc:374] T 00000000000000000000000000000000 P c88b3807d6bd413bb876555a253b8680 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 06:26:49.062963 31354 raft_consensus.cc:729] T 00000000000000000000000000000000 P c88b3807d6bd413bb876555a253b8680 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c88b3807d6bd413bb876555a253b8680, State: Initialized, Role: FOLLOWER
I20260806 06:26:49.063061 31354 consensus_queue.cc:260] T 00000000000000000000000000000000 P c88b3807d6bd413bb876555a253b8680 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2589a7a56e404deda1060eb408ebf54c" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 41635 } } peers { permanent_uuid: "124d842e9e484f8f85ddbc038dc51a31" member_type: VOTER last_known_addr { host: "127.21.12.189" port: 35823 } } peers { permanent_uuid: "c88b3807d6bd413bb876555a253b8680" member_type: VOTER last_known_addr { host: "127.21.12.188" port: 35289 } }
I20260806 06:26:49.063364 31354 sys_catalog.cc:564] T 00000000000000000000000000000000 P c88b3807d6bd413bb876555a253b8680 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 06:26:49.063340 31362 sys_catalog.cc:455] T 00000000000000000000000000000000 P c88b3807d6bd413bb876555a253b8680 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2589a7a56e404deda1060eb408ebf54c" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 41635 } } peers { permanent_uuid: "124d842e9e484f8f85ddbc038dc51a31" member_type: VOTER last_known_addr { host: "127.21.12.189" port: 35823 } } peers { permanent_uuid: "c88b3807d6bd413bb876555a253b8680" member_type: VOTER last_known_addr { host: "127.21.12.188" port: 35289 } } }
I20260806 06:26:49.063408 31362 sys_catalog.cc:458] T 00000000000000000000000000000000 P c88b3807d6bd413bb876555a253b8680 [sys.catalog]: This master's current role is: FOLLOWER
W20260806 06:26:49.065529 31378 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c88b3807d6bd413bb876555a253b8680: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 06:26:49.065608 31378 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 06:26:49.067649 31242 master_runner.cc:428] Detected that this master c88b3807d6bd413bb876555a253b8680 is joining an existing cluster
I20260806 06:26:49.067693 31242 master_runner.cc:432] Initiating AddMaster RPC to add 127.21.12.188:35289
I20260806 06:26:49.075816 30277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:49.068771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50766 (local address 127.21.12.190:41635)
0806 06:26:49.068818 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0806 06:26:49.068820 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:26:49.069157 (+ 337us) server_negotiation.cc:407] Connection header received
0806 06:26:49.069279 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:26:49.069280 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:26:49.069292 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:26:49.069348 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:26:49.069807 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:49.070597 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:49.071906 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:49.072146 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:49.072212 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:49.073260 (+ 1048us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:26:49.073261 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:26:49.073261 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 06:26:49.073268 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:26:49.074604 (+ 1336us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:26:49.075025 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:26:49.075026 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:26:49.075027 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:26:49.075089 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:26:49.075274 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:26:49.075275 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:26:49.075275 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:26:49.075397 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:26:49.075435 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0806 06:26:49.075635 (+ 200us) server_negotiation.cc:299] Negotiation successful
0806 06:26:49.075709 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260806 06:26:49.076185 31386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:49.068757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:41635 (local address 127.0.0.1:50766)
0806 06:26:49.069020 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0806 06:26:49.069031 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0806 06:26:49.069186 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:26:49.069353 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:26:49.069354 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:26:49.069566 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0806 06:26:49.069721 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:49.069724 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:49.070697 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:49.070698 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:49.071812 (+ 1114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:49.071816 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:49.072223 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:49.072225 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:49.072294 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:49.072679 (+ 385us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 06:26:49.072683 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0806 06:26:49.073170 (+ 487us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 06:26:49.074678 (+ 1508us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:26:49.074679 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:26:49.074679 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:26:49.074937 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:26:49.075155 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:26:49.075155 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:26:49.075155 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:26:49.075212 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:26:49.075474 (+ 262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 06:26:49.075475 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 06:26:49.075525 (+ 50us) client_negotiation.cc:769] Sending connection context
0806 06:26:49.075980 (+ 455us) client_negotiation.cc:240] Negotiation successful
0806 06:26:49.076081 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":147,"threads_started":1}
Aug 06 06:26:49 dist-test-slave-4tdj krb5kdc[30146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997608, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.12.190@KRBTEST.COM for oryx/127.21.12.188@KRBTEST.COM
I20260806 06:26:49.093618 31239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:49.069119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59474 (local address 127.21.12.189:35823)
0806 06:26:49.069176 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0806 06:26:49.069178 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:26:49.069392 (+ 214us) server_negotiation.cc:407] Connection header received
0806 06:26:49.069483 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:26:49.069484 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:26:49.069495 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:26:49.069561 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:26:49.070038 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:49.070575 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:49.071335 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:49.071559 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:49.071619 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:49.090732 (+ 19113us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:26:49.090735 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:26:49.090735 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 06:26:49.090743 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:26:49.092325 (+ 1582us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:26:49.092771 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:26:49.092773 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:26:49.092773 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:26:49.092838 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:26:49.093046 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:26:49.093047 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:26:49.093047 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:26:49.093186 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:26:49.093233 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0806 06:26:49.093429 (+ 196us) server_negotiation.cc:299] Negotiation successful
0806 06:26:49.093511 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260806 06:26:49.093946 31388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:49.068819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.189:35823 (local address 127.0.0.1:59474)
0806 06:26:49.069336 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0806 06:26:49.069345 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0806 06:26:49.069415 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:26:49.069570 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:26:49.069571 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:26:49.069907 (+ 336us) client_negotiation.cc:189] Negotiated authn=SASL
0806 06:26:49.069966 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:49.069967 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:49.070637 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:49.070638 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:49.071262 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:49.071264 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:49.071676 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:49.071677 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:49.071760 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:49.072159 (+ 399us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 06:26:49.072164 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0806 06:26:49.090648 (+ 18484us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 06:26:49.092406 (+ 1758us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:26:49.092407 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:26:49.092407 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:26:49.092680 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:26:49.092912 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:26:49.092913 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:26:49.092913 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:26:49.092972 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:26:49.093279 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 06:26:49.093280 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 06:26:49.093320 (+ 40us) client_negotiation.cc:769] Sending connection context
0806 06:26:49.093710 (+ 390us) client_negotiation.cc:240] Negotiation successful
0806 06:26:49.093838 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":500,"thread_start_us":178,"threads_started":1}
I20260806 06:26:49.095008 31103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:49.078259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.188:35289 (local address 127.0.0.1:35312)
0806 06:26:49.078316 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0806 06:26:49.078326 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0806 06:26:49.078365 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:26:49.078872 (+ 507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:26:49.078873 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:26:49.078985 (+ 112us) client_negotiation.cc:189] Negotiated authn=SASL
0806 06:26:49.079018 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:49.079020 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:49.079988 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:49.079989 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:49.080453 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:49.080454 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:49.081000 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:49.081001 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:49.081061 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:49.081412 (+ 351us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 06:26:49.081415 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0806 06:26:49.082640 (+ 1225us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 06:26:49.093350 (+ 10710us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:26:49.093352 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:26:49.093352 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:26:49.093579 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:26:49.094368 (+ 789us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:26:49.094369 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:26:49.094369 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:26:49.094433 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:26:49.094712 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 06:26:49.094714 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 06:26:49.094760 (+ 46us) client_negotiation.cc:769] Sending connection context
0806 06:26:49.094824 (+ 64us) client_negotiation.cc:240] Negotiation successful
0806 06:26:49.094920 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24}
I20260806 06:26:49.095033 31356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:49.069061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35306 (local address 127.21.12.188:35289)
0806 06:26:49.069092 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0806 06:26:49.069094 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:26:49.074472 (+ 5378us) server_negotiation.cc:407] Connection header received
0806 06:26:49.074554 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:26:49.074555 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:26:49.074563 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:26:49.074588 (+ 25us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:26:49.075114 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:49.075721 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:49.076804 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:49.077080 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:49.077116 (+ 36us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:49.091733 (+ 14617us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:26:49.091736 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:26:49.091737 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 06:26:49.091745 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:26:49.093593 (+ 1848us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:26:49.094007 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:26:49.094008 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:26:49.094008 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:26:49.094064 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:26:49.094198 (+ 134us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:26:49.094198 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:26:49.094199 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:26:49.094324 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:26:49.094364 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0806 06:26:49.094776 (+ 412us) server_negotiation.cc:299] Negotiation successful
0806 06:26:49.094908 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":56,"server-negotiator.queue_time_us":17}
I20260806 06:26:49.095284 31387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:49.069173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.188:35289 (local address 127.0.0.1:35306)
0806 06:26:49.074408 (+ 5235us) negotiation.cc:107] Waiting for socket to connect
0806 06:26:49.074419 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0806 06:26:49.074495 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:26:49.074634 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:26:49.074635 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:26:49.074933 (+ 298us) client_negotiation.cc:189] Negotiated authn=SASL
0806 06:26:49.075027 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:49.075029 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:49.075793 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:49.075794 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:49.076726 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:49.076729 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:49.077289 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:49.077289 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:49.077363 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:49.090996 (+ 13633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 06:26:49.091001 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0806 06:26:49.091414 (+ 413us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 06:26:49.093663 (+ 2249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:26:49.093664 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:26:49.093664 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:26:49.093929 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:26:49.094113 (+ 184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:26:49.094113 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:26:49.094113 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:26:49.094148 (+ 35us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:26:49.094563 (+ 415us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 06:26:49.094564 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 06:26:49.094660 (+ 96us) client_negotiation.cc:769] Sending connection context
0806 06:26:49.094748 (+ 88us) client_negotiation.cc:240] Negotiation successful
0806 06:26:49.094850 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5213,"thread_start_us":107,"threads_started":1}
I20260806 06:26:49.095389 31389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:49.078323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35312 (local address 127.21.12.188:35289)
0806 06:26:49.078716 (+ 393us) server_negotiation.cc:206] Beginning negotiation
0806 06:26:49.078718 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:26:49.078726 (+ 8us) server_negotiation.cc:407] Connection header received
0806 06:26:49.078762 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:26:49.078763 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:26:49.078774 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:26:49.078843 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:26:49.079089 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:49.079913 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:49.080525 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:49.080900 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:49.080973 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:49.082850 (+ 1877us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:26:49.082853 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:26:49.082853 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 06:26:49.082860 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:26:49.093241 (+ 10381us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:26:49.093883 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:26:49.093884 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:26:49.093884 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:26:49.094310 (+ 426us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:26:49.094499 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:26:49.094500 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:26:49.094500 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:26:49.094648 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:26:49.094718 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0806 06:26:49.095189 (+ 471us) server_negotiation.cc:299] Negotiation successful
0806 06:26:49.095269 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":303,"thread_start_us":257,"threads_started":1}
I20260806 06:26:49.096148 21554 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHhLhT/build/release/bin/kudu with pid 30313
I20260806 06:26:49.096122 30217 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.21.12.188:35289: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "c88b3807d6bd413bb876555a253b8680"
member_type: NON_VOTER
last_known_addr {
host: "127.21.12.188"
port: 35289
}
attrs {
promote: true
}
}
dest_uuid: "2589a7a56e404deda1060eb408ebf54c"
cas_config_opid_index: 5
I20260806 06:26:49.096356 30217 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2589a7a56e404deda1060eb408ebf54c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "2589a7a56e404deda1060eb408ebf54c" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 41635 } } peers { permanent_uuid: "124d842e9e484f8f85ddbc038dc51a31" member_type: NON_VOTER last_known_addr { host: "127.21.12.189" port: 35823 } attrs { promote: true } } peers { permanent_uuid: "c88b3807d6bd413bb876555a253b8680" member_type: NON_VOTER last_known_addr { host: "127.21.12.188" port: 35289 } attrs { promote: true } }
I20260806 06:26:49.098906 31295 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c88b3807d6bd413bb876555a253b8680 [term 0 FOLLOWER]: Advancing to term 1
I20260806 06:26:49.100175 31295 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c88b3807d6bd413bb876555a253b8680 [term 1 FOLLOWER]: Refusing update from remote peer 2589a7a56e404deda1060eb408ebf54c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260806 06:26:49.100422 31107 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2589a7a56e404deda1060eb408ebf54c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c88b3807d6bd413bb876555a253b8680" member_type: NON_VOTER last_known_addr { host: "127.21.12.188" port: 35289 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 06:26:49.102381 31295 raft_consensus.cc:858] T 00000000000000000000000000000000 P c88b3807d6bd413bb876555a253b8680 [term 1 NON_PARTICIPANT]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2589a7a56e404deda1060eb408ebf54c" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 41635 } } peers { permanent_uuid: "124d842e9e484f8f85ddbc038dc51a31" member_type: VOTER last_known_addr { host: "127.21.12.189" port: 35823 } } peers { permanent_uuid: "c88b3807d6bd413bb876555a253b8680" member_type: VOTER last_known_addr { host: "127.21.12.188" port: 35289 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "2589a7a56e404deda1060eb408ebf54c" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 41635 } } peers { permanent_uuid: "124d842e9e484f8f85ddbc038dc51a31" member_type: NON_VOTER last_known_addr { host: "127.21.12.189" port: 35823 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "2589a7a56e404deda1060eb408ebf54c" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 41635 } } peers { permanent_uuid: "124d842e9e484f8f85ddbc038dc51a31" member_type: NON_VOTER last_known_addr { host: "127.21.12.189" port: 35823 } attrs { promote: true } } peers { permanent_uuid: "c88b3807d6bd413bb876555a253b8680" member_type: NON_VOTER last_known_addr { host: "127.21.12.188" port: 35289 } attrs { promote: true } }
W20260806 06:26:49.102568 31295 raft_consensus.cc:1565] T 00000000000000000000000000000000 P c88b3807d6bd413bb876555a253b8680 [term 1 NON_PARTICIPANT]: Could not prepare op 'term: 1 index: 6' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2589a7a56e404deda1060eb408ebf54c" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 41635 } } peers { permanent_uuid: "124d842e9e484f8f85ddbc038dc51a31" member_type: VOTER last_known_addr { host: "127.21.12.189" port: 35823 } } peers { permanent_uuid: "c88b3807d6bd413bb876555a253b8680" member_type: VOTER last_known_addr { host: "127.21.12.188" port: 35289 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "2589a7a56e404deda1060eb408ebf54c" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 41635 } } peers { permanent_uuid: "124d842e9e484f8f85ddbc038dc51a31" member_type: NON_VOTER last_known_addr { host: "127.21.12.189" port: 35823 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "2589a7a56e404deda1060eb408ebf54c" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 41635 } } peers { permanent_uuid: "124d842e9e484f8f85ddbc038dc51a31" member_type: NON_VOTER last_known_addr { host: "127.21.12.189" port: 35823 } attrs { promote: true } } peers { permanent_uuid: "c88b3807d6bd413bb876555a253b8680" member_type: NON_VOTER last_known_addr { host: "127.21.12.188" port: 35289 } attrs { promote: true } }
I20260806 06:26:49.104372 31295 raft_consensus.cc:2932] T 00000000000000000000000000000000 P c88b3807d6bd413bb876555a253b8680 [term 1 NON_PARTICIPANT]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 124d842e9e484f8f85ddbc038dc51a31 (127.21.12.189) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "2589a7a56e404deda1060eb408ebf54c" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 41635 } } peers { permanent_uuid: "124d842e9e484f8f85ddbc038dc51a31" member_type: NON_VOTER last_known_addr { host: "127.21.12.189" port: 35823 } attrs { promote: true } } }
I20260806 06:26:49.106082 31362 sys_catalog.cc:455] T 00000000000000000000000000000000 P c88b3807d6bd413bb876555a253b8680 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2589a7a56e404deda1060eb408ebf54c. Latest consensus state: current_term: 1 leader_uuid: "2589a7a56e404deda1060eb408ebf54c" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "2589a7a56e404deda1060eb408ebf54c" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 41635 } } peers { permanent_uuid: "124d842e9e484f8f85ddbc038dc51a31" member_type: NON_VOTER last_known_addr { host: "127.21.12.189" port: 35823 } attrs { promote: true } } }
I20260806 06:26:49.106070 31107 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2589a7a56e404deda1060eb408ebf54c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2589a7a56e404deda1060eb408ebf54c" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "2589a7a56e404deda1060eb408ebf54c" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 41635 } } peers { permanent_uuid: "124d842e9e484f8f85ddbc038dc51a31" member_type: NON_VOTER last_known_addr { host: "127.21.12.189" port: 35823 } attrs { promote: true } } } pending_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "2589a7a56e404deda1060eb408ebf54c" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 41635 } } peers { permanent_uuid: "124d842e9e484f8f85ddbc038dc51a31" member_type: NON_VOTER last_known_addr { host: "127.21.12.189" port: 35823 } attrs { promote: true } } peers { permanent_uuid: "c88b3807d6bd413bb876555a253b8680" member_type: NON_VOTER last_known_addr { host: "127.21.12.188" port: 35289 } attrs { promote: true } } }
I20260806 06:26:49.106153 31362 sys_catalog.cc:458] T 00000000000000000000000000000000 P c88b3807d6bd413bb876555a253b8680 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260806 06:26:49.106163 31107 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2589a7a56e404deda1060eb408ebf54c [sys.catalog]: This master's current role is: LEADER
I20260806 06:26:49.106185 31362 sys_catalog.cc:455] T 00000000000000000000000000000000 P c88b3807d6bd413bb876555a253b8680 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2589a7a56e404deda1060eb408ebf54c" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "2589a7a56e404deda1060eb408ebf54c" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 41635 } } peers { permanent_uuid: "124d842e9e484f8f85ddbc038dc51a31" member_type: NON_VOTER last_known_addr { host: "127.21.12.189" port: 35823 } attrs { promote: true } } }
I20260806 06:26:49.106220 31362 sys_catalog.cc:458] T 00000000000000000000000000000000 P c88b3807d6bd413bb876555a253b8680 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260806 06:26:49.108170 31393 mvcc.cc:205] Tried to move back new op lower bound from 7315446202782715904 to 7315446202707832832. Current Snapshot: MvccSnapshot[applied={T|T < 7315446202782715904}]
I20260806 06:26:49.111173 31296 raft_consensus.cc:1467] T 00000000000000000000000000000000 P c88b3807d6bd413bb876555a253b8680 [term 1 NON_PARTICIPANT]: Allowing update even though not a member of the config
I20260806 06:26:49.111366 31395 sys_catalog.cc:455] T 00000000000000000000000000000000 P c88b3807d6bd413bb876555a253b8680 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2589a7a56e404deda1060eb408ebf54c" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "2589a7a56e404deda1060eb408ebf54c" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 41635 } } peers { permanent_uuid: "124d842e9e484f8f85ddbc038dc51a31" member_type: NON_VOTER last_known_addr { host: "127.21.12.189" port: 35823 } attrs { promote: true } } }
I20260806 06:26:49.111430 31395 sys_catalog.cc:458] T 00000000000000000000000000000000 P c88b3807d6bd413bb876555a253b8680 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260806 06:26:49.111899 31106 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 2589a7a56e404deda1060eb408ebf54c [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER c88b3807d6bd413bb876555a253b8680 (127.21.12.188) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "2589a7a56e404deda1060eb408ebf54c" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 41635 } } peers { permanent_uuid: "124d842e9e484f8f85ddbc038dc51a31" member_type: NON_VOTER last_known_addr { host: "127.21.12.189" port: 35823 } attrs { promote: true } } peers { permanent_uuid: "c88b3807d6bd413bb876555a253b8680" member_type: NON_VOTER last_known_addr { host: "127.21.12.188" port: 35289 } attrs { promote: true } } }
I20260806 06:26:49.112994 31106 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.21.12.188:35289
I20260806 06:26:49.113085 31106 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2589a7a56e404deda1060eb408ebf54c [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "2589a7a56e404deda1060eb408ebf54c" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "2589a7a56e404deda1060eb408ebf54c" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 41635 } } peers { permanent_uuid: "124d842e9e484f8f85ddbc038dc51a31" member_type: NON_VOTER last_known_addr { host: "127.21.12.189" port: 35823 } attrs { promote: true } } peers { permanent_uuid: "c88b3807d6bd413bb876555a253b8680" member_type: NON_VOTER last_known_addr { host: "127.21.12.188" port: 35289 } attrs { promote: true } } }
I20260806 06:26:49.113143 31106 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2589a7a56e404deda1060eb408ebf54c [sys.catalog]: This master's current role is: LEADER
I20260806 06:26:49.113363 31242 master.cc:537] Master@127.21.12.188:35289 shutting down...
I20260806 06:26:49.113987 31107 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 2589a7a56e404deda1060eb408ebf54c: attempting to promote NON_VOTER c88b3807d6bd413bb876555a253b8680 to VOTER
I20260806 06:26:49.114172 31107 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2589a7a56e404deda1060eb408ebf54c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2589a7a56e404deda1060eb408ebf54c" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 41635 } } peers { permanent_uuid: "124d842e9e484f8f85ddbc038dc51a31" member_type: NON_VOTER last_known_addr { host: "127.21.12.189" port: 35823 } attrs { promote: true } } peers { permanent_uuid: "c88b3807d6bd413bb876555a253b8680" member_type: VOTER last_known_addr { host: "127.21.12.188" port: 35289 } attrs { promote: false } }
I20260806 06:26:49.114928 31103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:49.096822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.189:35823 (local address 127.0.0.1:59476)
0806 06:26:49.098745 (+ 1923us) negotiation.cc:107] Waiting for socket to connect
0806 06:26:49.098758 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 06:26:49.098801 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:26:49.111104 (+ 12303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:26:49.111105 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:26:49.111108 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 06:26:49.111157 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:49.111160 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:49.112488 (+ 1328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:49.112490 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:49.113632 (+ 1142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:49.113636 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:49.114683 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:49.114684 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:49.114749 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:49.114771 (+ 22us) client_negotiation.cc:769] Sending connection context
0806 06:26:49.114821 (+ 50us) client_negotiation.cc:240] Negotiation successful
0806 06:26:49.114864 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1903}
I20260806 06:26:49.115245 31239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:49.110841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59476 (local address 127.21.12.189:35823)
0806 06:26:49.110890 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0806 06:26:49.110892 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:26:49.110903 (+ 11us) server_negotiation.cc:407] Connection header received
0806 06:26:49.110930 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:26:49.110932 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:26:49.110957 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:26:49.111033 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 06:26:49.111555 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:49.112311 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:49.113838 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:49.114416 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:49.114483 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:49.114577 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0806 06:26:49.115137 (+ 560us) server_negotiation.cc:299] Negotiation successful
0806 06:26:49.115182 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260806 06:26:49.115481 31182 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 124d842e9e484f8f85ddbc038dc51a31 [term 1 LEARNER]: Refusing update from remote peer 2589a7a56e404deda1060eb408ebf54c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260806 06:26:49.115581 31181 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 124d842e9e484f8f85ddbc038dc51a31 [term 1 LEARNER]: Refusing update from remote peer 2589a7a56e404deda1060eb408ebf54c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20260806 06:26:49.115731 30192 consensus_peers.cc:487] T 00000000000000000000000000000000 P 2589a7a56e404deda1060eb408ebf54c -> Peer c88b3807d6bd413bb876555a253b8680 (127.21.12.188:35289): Couldn't send request to peer c88b3807d6bd413bb876555a253b8680. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260806 06:26:49.116710 31400 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2589a7a56e404deda1060eb408ebf54c [LEADER]: Connected to new peer: Peer: permanent_uuid: "124d842e9e484f8f85ddbc038dc51a31" member_type: NON_VOTER last_known_addr { host: "127.21.12.189" port: 35823 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20260806 06:26:49.117162 31181 raft_consensus.cc:858] T 00000000000000000000000000000000 P 124d842e9e484f8f85ddbc038dc51a31 [term 1 LEARNER]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "2589a7a56e404deda1060eb408ebf54c" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 41635 } } peers { permanent_uuid: "124d842e9e484f8f85ddbc038dc51a31" member_type: NON_VOTER last_known_addr { host: "127.21.12.189" port: 35823 } attrs { promote: true } }.
Pending config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "2589a7a56e404deda1060eb408ebf54c" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 41635 } } peers { permanent_uuid: "124d842e9e484f8f85ddbc038dc51a31" member_type: NON_VOTER last_known_addr { host: "127.21.12.189" port: 35823 } attrs { promote: true } } peers { permanent_uuid: "c88b3807d6bd413bb876555a253b8680" member_type: NON_VOTER last_known_addr { host: "127.21.12.188" port: 35289 } attrs { promote: true } }:
New config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2589a7a56e404deda1060eb408ebf54c" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 41635 } } peers { permanent_uuid: "124d842e9e484f8f85ddbc038dc51a31" member_type: NON_VOTER last_known_addr { host: "127.21.12.189" port: 35823 } attrs { promote: true } } peers { permanent_uuid: "c88b3807d6bd413bb876555a253b8680" member_type: VOTER last_known_addr { host: "127.21.12.188" port: 35289 } attrs { promote: false } }
W20260806 06:26:49.117223 31181 raft_consensus.cc:1565] T 00000000000000000000000000000000 P 124d842e9e484f8f85ddbc038dc51a31 [term 1 LEARNER]: Could not prepare op 'term: 1 index: 7' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "2589a7a56e404deda1060eb408ebf54c" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 41635 } } peers { permanent_uuid: "124d842e9e484f8f85ddbc038dc51a31" member_type: NON_VOTER last_known_addr { host: "127.21.12.189" port: 35823 } attrs { promote: true } }.
Pending config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "2589a7a56e404deda1060eb408ebf54c" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 41635 } } peers { permanent_uuid: "124d842e9e484f8f85ddbc038dc51a31" member_type: NON_VOTER last_known_addr { host: "127.21.12.189" port: 35823 } attrs { promote: true } } peers { permanent_uuid: "c88b3807d6bd413bb876555a253b8680" member_type: NON_VOTER last_known_addr { host: "127.21.12.188" port: 35289 } attrs { promote: true } }:
New config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2589a7a56e404deda1060eb408ebf54c" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 41635 } } peers { permanent_uuid: "124d842e9e484f8f85ddbc038dc51a31" member_type: NON_VOTER last_known_addr { host: "127.21.12.189" port: 35823 } attrs { promote: true } } peers { permanent_uuid: "c88b3807d6bd413bb876555a253b8680" member_type: VOTER last_known_addr { host: "127.21.12.188" port: 35289 } attrs { promote: false } }
I20260806 06:26:49.117434 31181 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 124d842e9e484f8f85ddbc038dc51a31 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER c88b3807d6bd413bb876555a253b8680 (127.21.12.188) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "2589a7a56e404deda1060eb408ebf54c" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 41635 } } peers { permanent_uuid: "124d842e9e484f8f85ddbc038dc51a31" member_type: NON_VOTER last_known_addr { host: "127.21.12.189" port: 35823 } attrs { promote: true } } peers { permanent_uuid: "c88b3807d6bd413bb876555a253b8680" member_type: NON_VOTER last_known_addr { host: "127.21.12.188" port: 35289 } attrs { promote: true } } }
I20260806 06:26:49.118575 31223 sys_catalog.cc:455] T 00000000000000000000000000000000 P 124d842e9e484f8f85ddbc038dc51a31 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2589a7a56e404deda1060eb408ebf54c. Latest consensus state: current_term: 1 leader_uuid: "2589a7a56e404deda1060eb408ebf54c" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "2589a7a56e404deda1060eb408ebf54c" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 41635 } } peers { permanent_uuid: "124d842e9e484f8f85ddbc038dc51a31" member_type: NON_VOTER last_known_addr { host: "127.21.12.189" port: 35823 } attrs { promote: true } } peers { permanent_uuid: "c88b3807d6bd413bb876555a253b8680" member_type: NON_VOTER last_known_addr { host: "127.21.12.188" port: 35289 } attrs { promote: true } } }
I20260806 06:26:49.118659 31223 sys_catalog.cc:458] T 00000000000000000000000000000000 P 124d842e9e484f8f85ddbc038dc51a31 [sys.catalog]: This master's current role is: LEARNER
I20260806 06:26:49.118686 31223 sys_catalog.cc:455] T 00000000000000000000000000000000 P 124d842e9e484f8f85ddbc038dc51a31 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2589a7a56e404deda1060eb408ebf54c" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "2589a7a56e404deda1060eb408ebf54c" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 41635 } } peers { permanent_uuid: "124d842e9e484f8f85ddbc038dc51a31" member_type: NON_VOTER last_known_addr { host: "127.21.12.189" port: 35823 } attrs { promote: true } } peers { permanent_uuid: "c88b3807d6bd413bb876555a253b8680" member_type: NON_VOTER last_known_addr { host: "127.21.12.188" port: 35289 } attrs { promote: true } } }
I20260806 06:26:49.118716 31223 sys_catalog.cc:458] T 00000000000000000000000000000000 P 124d842e9e484f8f85ddbc038dc51a31 [sys.catalog]: This master's current role is: LEARNER
I20260806 06:26:49.118937 31242 raft_consensus.cc:2229] T 00000000000000000000000000000000 P c88b3807d6bd413bb876555a253b8680 [term 1 LEARNER]: Raft consensus shutting down.
I20260806 06:26:49.119065 31242 pending_rounds.cc:62] T 00000000000000000000000000000000 P c88b3807d6bd413bb876555a253b8680: Trying to abort 1 pending ops.
I20260806 06:26:49.119052 21554 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHhLhT/build/release/bin/kudu
/tmp/dist-test-taskFHhLhT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785997581049705-21554-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785997581049705-21554-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785997581049705-21554-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785997581049705-21554-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.12.129:41895
--local_ip_for_outbound_sockets=127.21.12.129
--tserver_master_addrs=127.21.12.190:41635
--webserver_port=35087
--webserver_interface=127.21.12.129
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785997581049705-21554-0/krb5kdc/oryx_127.21.12.129.keytab
--principal=oryx/127.21.12.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.148:39091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.21.12.190:41635,127.21.12.189:35823
--tserver_master_addrs=127.21.12.190:41635,127.21.12.189:35823,127.21.12.188:35289 with env {KRB5CCNAME=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785997581049705-21554-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785997581049705-21554-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785997581049705-21554-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 06:26:49.119098 31242 pending_rounds.cc:68] T 00000000000000000000000000000000 P c88b3807d6bd413bb876555a253b8680: Aborting op as it isn't in flight: id { term: 1 index: 6 } timestamp: 7315446206858117120 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "00000000000000000000000000000000" old_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "2589a7a56e404deda1060eb408ebf54c" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 41635 } } peers { permanent_uuid: "124d842e9e484f8f85ddbc038dc51a31" member_type: NON_VOTER last_known_addr { host: "127.21.12.189" port: 35823 } attrs { promote: true } } } new_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "2589a7a56e404deda1060eb408ebf54c" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 41635 } } peers { permanent_uuid: "124d842e9e484f8f85ddbc038dc51a31" member_type: NON_VOTER last_known_addr { host: "127.21.12.189" port: 35823 } attrs { promote: true } } peers { permanent_uuid: "c88b3807d6bd413bb876555a253b8680" member_type: NON_VOTER last_known_addr { host: "127.21.12.188" port: 35289 } attrs { promote: true } } } }
I20260806 06:26:49.119189 31242 raft_consensus.cc:2894] T 00000000000000000000000000000000 P c88b3807d6bd413bb876555a253b8680 [term 1 LEARNER]: Aborting config change with OpId 1.6: Aborted: Op aborted
I20260806 06:26:49.119225 31242 raft_consensus.cc:2866] T 00000000000000000000000000000000 P c88b3807d6bd413bb876555a253b8680 [term 1 NON_PARTICIPANT]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20260806 06:26:49.119259 31242 raft_consensus.cc:2258] T 00000000000000000000000000000000 P c88b3807d6bd413bb876555a253b8680 [term 1 NON_PARTICIPANT]: Raft consensus is shut down!
I20260806 06:26:49.119288 31242 tablet_replica.cc:330] T 00000000000000000000000000000000 P c88b3807d6bd413bb876555a253b8680: stopping tablet replica
I20260806 06:26:49.120247 31106 raft_consensus.cc:1013] T 00000000000000000000000000000000 P 2589a7a56e404deda1060eb408ebf54c [term 1 LEADER]: attempt to promote peer 124d842e9e484f8f85ddbc038dc51a31: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20260806 06:26:49.120334 31106 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2589a7a56e404deda1060eb408ebf54c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2589a7a56e404deda1060eb408ebf54c" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "2589a7a56e404deda1060eb408ebf54c" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 41635 } } peers { permanent_uuid: "124d842e9e484f8f85ddbc038dc51a31" member_type: NON_VOTER last_known_addr { host: "127.21.12.189" port: 35823 } attrs { promote: true } } peers { permanent_uuid: "c88b3807d6bd413bb876555a253b8680" member_type: NON_VOTER last_known_addr { host: "127.21.12.188" port: 35289 } attrs { promote: true } } } pending_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2589a7a56e404deda1060eb408ebf54c" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 41635 } } peers { permanent_uuid: "124d842e9e484f8f85ddbc038dc51a31" member_type: NON_VOTER last_known_addr { host: "127.21.12.189" port: 35823 } attrs { promote: true } } peers { permanent_uuid: "c88b3807d6bd413bb876555a253b8680" member_type: VOTER last_known_addr { host: "127.21.12.188" port: 35289 } attrs { promote: false } } }
I20260806 06:26:49.120412 31106 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2589a7a56e404deda1060eb408ebf54c [sys.catalog]: This master's current role is: LEADER
I20260806 06:26:49.120824 31107 raft_consensus.cc:1013] T 00000000000000000000000000000000 P 2589a7a56e404deda1060eb408ebf54c [term 1 LEADER]: attempt to promote peer 124d842e9e484f8f85ddbc038dc51a31: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20260806 06:26:49.123651 31242 master.cc:552] Master@127.21.12.188:35289 shutdown complete.
I20260806 06:26:49.123731 31242 master_runner.cc:305] Clearing existing system tablet
I20260806 06:26:49.123893 31242 fs_manager.cc:359] Metadata directory not provided
I20260806 06:26:49.123968 31242 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 06:26:49.124742 31242 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.000s
I20260806 06:26:49.125362 31406 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 06:26:49.125527 31242 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785997581049705-21554-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 06:26:49.125699 31242 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260806 06:26:49.125764 31242 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785997581049705-21554-0/minicluster-data/master-2/data,/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785997581049705-21554-0/minicluster-data/master-2/wal
uuid: "c88b3807d6bd413bb876555a253b8680"
format_stamp: "Formatted at 2026-08-06 06:26:48 on dist-test-slave-4tdj"
server_key: "0147de3e322a27788c3f07ea29117340"
server_key_iv: "523e6807d20dabc92705fbdee875d122"
server_key_version: "encryptionkey@0"
I20260806 06:26:49.126606 31242 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P c88b3807d6bd413bb876555a253b8680: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 06:26:49.128053 31242 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P c88b3807d6bd413bb876555a253b8680: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260806 06:26:49.128193 31242 log.cc:1196] T 00000000000000000000000000000000 P c88b3807d6bd413bb876555a253b8680: Deleting WAL directory at /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785997581049705-21554-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260806 06:26:49.128486 31242 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P c88b3807d6bd413bb876555a253b8680: Deleting consensus metadata
I20260806 06:26:49.128654 31242 master_runner.cc:315] Copying system tablet from 127.21.12.190:41635
I20260806 06:26:49.129294 31242 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P c88b3807d6bd413bb876555a253b8680: tablet copy: Beginning tablet copy session from remote peer at address 127.21.12.190:41635
I20260806 06:26:49.136835 30277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:49.129720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50780 (local address 127.21.12.190:41635)
0806 06:26:49.129769 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0806 06:26:49.129771 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:26:49.129887 (+ 116us) server_negotiation.cc:407] Connection header received
0806 06:26:49.129948 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:26:49.129949 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:26:49.129960 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:26:49.130027 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:26:49.130699 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:49.131471 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:49.132512 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:49.132819 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:49.132894 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:49.133971 (+ 1077us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:26:49.133973 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:26:49.133973 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 06:26:49.133981 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:26:49.135655 (+ 1674us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:26:49.136051 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:26:49.136053 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:26:49.136053 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:26:49.136124 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:26:49.136321 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:26:49.136322 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:26:49.136322 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:26:49.136448 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:26:49.136511 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0806 06:26:49.136662 (+ 151us) server_negotiation.cc:299] Negotiation successful
0806 06:26:49.136741 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260806 06:26:49.136843 31412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:49.129671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:41635 (local address 127.0.0.1:50780)
0806 06:26:49.129813 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0806 06:26:49.129824 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0806 06:26:49.129893 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:26:49.130042 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:26:49.130043 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:26:49.130544 (+ 501us) client_negotiation.cc:189] Negotiated authn=SASL
0806 06:26:49.130617 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:49.130619 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:49.131560 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:49.131561 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:49.132432 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:49.132434 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:49.132892 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:49.132892 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:49.132960 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:49.133451 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 06:26:49.133456 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0806 06:26:49.133873 (+ 417us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 06:26:49.135729 (+ 1856us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:26:49.135731 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:26:49.135731 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:26:49.135973 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:26:49.136191 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:26:49.136192 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:26:49.136192 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:26:49.136262 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:26:49.136508 (+ 246us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 06:26:49.136509 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 06:26:49.136566 (+ 57us) client_negotiation.cc:769] Sending connection context
0806 06:26:49.136630 (+ 64us) client_negotiation.cc:240] Negotiation successful
0806 06:26:49.136735 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":68,"threads_started":1}
I20260806 06:26:49.137204 30250 tablet_copy_service.cc:140] P 2589a7a56e404deda1060eb408ebf54c: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer c88b3807d6bd413bb876555a253b8680 ({username='oryx', principal='oryx/127.21.12.188@KRBTEST.COM'} at 127.0.0.1:50780)
I20260806 06:26:49.137266 30250 tablet_copy_service.cc:161] P 2589a7a56e404deda1060eb408ebf54c: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer c88b3807d6bd413bb876555a253b8680 at {username='oryx', principal='oryx/127.21.12.188@KRBTEST.COM'} at 127.0.0.1:50780: session id = c88b3807d6bd413bb876555a253b8680-00000000000000000000000000000000
I20260806 06:26:49.137848 30250 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 2589a7a56e404deda1060eb408ebf54c: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 06:26:49.138425 31242 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260806 06:26:49.139696 31242 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P c88b3807d6bd413bb876555a253b8680: tablet copy: Starting download of 0 data blocks...
I20260806 06:26:49.139918 31242 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P c88b3807d6bd413bb876555a253b8680: tablet copy: Starting download of 1 WAL segments...
I20260806 06:26:49.140769 31242 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P c88b3807d6bd413bb876555a253b8680: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 06:26:49.141690 30250 tablet_copy_service.cc:342] P 2589a7a56e404deda1060eb408ebf54c: Request end of tablet copy session c88b3807d6bd413bb876555a253b8680-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.21.12.188@KRBTEST.COM'} at 127.0.0.1:50780
I20260806 06:26:49.141750 30250 tablet_copy_service.cc:433] P 2589a7a56e404deda1060eb408ebf54c: ending tablet copy session c88b3807d6bd413bb876555a253b8680-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer c88b3807d6bd413bb876555a253b8680
I20260806 06:26:49.146924 31242 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 06:26:49.148176 31420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:26:49.148943 31242 server_base.cc:1034] running on GCE node
W20260806 06:26:49.150480 31414 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:26:49.154417 31413 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:26:49.154634 31242 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 06:26:49.154783 31242 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 06:26:49.156049 31242 hybrid_clock.cc:648] HybridClock initialized: now 1785997609156034 us; error 31 us; skew 500 ppm
Aug 06 06:26:49 dist-test-slave-4tdj krb5kdc[30146](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997609, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.12.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 06:26:49.157692 31242 init.cc:377] Logged in from keytab as oryx/127.21.12.188@KRBTEST.COM (short username oryx)
I20260806 06:26:49.157960 31242 webserver.cc:460] Webserver started at http://127.21.12.188:34413/ using document root <none> and password file <none>
I20260806 06:26:49.158079 31242 fs_manager.cc:359] Metadata directory not provided
I20260806 06:26:49.158108 31242 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 06:26:49.159116 31242 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.000s
I20260806 06:26:49.159687 31429 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 06:26:49.159878 31242 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260806 06:26:49.159942 31242 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785997581049705-21554-0/minicluster-data/master-2/data,/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785997581049705-21554-0/minicluster-data/master-2/wal
uuid: "c88b3807d6bd413bb876555a253b8680"
format_stamp: "Formatted at 2026-08-06 06:26:48 on dist-test-slave-4tdj"
server_key: "0147de3e322a27788c3f07ea29117340"
server_key_iv: "523e6807d20dabc92705fbdee875d122"
server_key_version: "encryptionkey@0"
I20260806 06:26:49.159993 31242 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785997581049705-21554-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785997581049705-21554-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785997581049705-21554-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 06:26:49.192226 31242 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 06:26:49.194619 31242 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 06:26:49.198442 31242 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.188:35289
I20260806 06:26:49.198485 31509 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.188:35289 every 8 connection(s)
I20260806 06:26:49.198961 31242 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785997581049705-21554-0/minicluster-data/master-2/data/info.pb
I20260806 06:26:49.199949 31510 sys_catalog.cc:263] Verifying existing consensus state
I20260806 06:26:49.200469 31510 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c88b3807d6bd413bb876555a253b8680: Bootstrap starting.
I20260806 06:26:49.203058 31510 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c88b3807d6bd413bb876555a253b8680: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=6 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20260806 06:26:49.203266 31510 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c88b3807d6bd413bb876555a253b8680: Bootstrap complete.
I20260806 06:26:49.203460 31510 raft_consensus.cc:348] T 00000000000000000000000000000000 P c88b3807d6bd413bb876555a253b8680 [term 1 LEARNER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "2589a7a56e404deda1060eb408ebf54c" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 41635 } } peers { permanent_uuid: "124d842e9e484f8f85ddbc038dc51a31" member_type: NON_VOTER last_known_addr { host: "127.21.12.189" port: 35823 } attrs { promote: true } } peers { permanent_uuid: "c88b3807d6bd413bb876555a253b8680" member_type: NON_VOTER last_known_addr { host: "127.21.12.188" port: 35289 } attrs { promote: true } }
I20260806 06:26:49.203621 31510 raft_consensus.cc:729] T 00000000000000000000000000000000 P c88b3807d6bd413bb876555a253b8680 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c88b3807d6bd413bb876555a253b8680, State: Initialized, Role: FOLLOWER
I20260806 06:26:49.203662 31510 consensus_queue.cc:260] T 00000000000000000000000000000000 P c88b3807d6bd413bb876555a253b8680 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2589a7a56e404deda1060eb408ebf54c" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 41635 } } peers { permanent_uuid: "124d842e9e484f8f85ddbc038dc51a31" member_type: NON_VOTER last_known_addr { host: "127.21.12.189" port: 35823 } attrs { promote: true } } peers { permanent_uuid: "c88b3807d6bd413bb876555a253b8680" member_type: VOTER last_known_addr { host: "127.21.12.188" port: 35289 } attrs { promote: false } }
I20260806 06:26:49.203821 31510 sys_catalog.cc:564] T 00000000000000000000000000000000 P c88b3807d6bd413bb876555a253b8680 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 06:26:49.205832 31532 catalog_manager.cc:1259] Loaded cluster ID: 2793aaf3eaab49c4806dfa641e5153fa
I20260806 06:26:49.205924 31532 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c88b3807d6bd413bb876555a253b8680: loading cluster ID for follower catalog manager: success
I20260806 06:26:49.207111 31532 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c88b3807d6bd413bb876555a253b8680: acquiring CA information for follower catalog manager: success
I20260806 06:26:49.207475 31532 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c88b3807d6bd413bb876555a253b8680: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260806 06:26:49.210391 31515 sys_catalog.cc:455] T 00000000000000000000000000000000 P c88b3807d6bd413bb876555a253b8680 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "2589a7a56e404deda1060eb408ebf54c" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 41635 } } peers { permanent_uuid: "124d842e9e484f8f85ddbc038dc51a31" member_type: NON_VOTER last_known_addr { host: "127.21.12.189" port: 35823 } attrs { promote: true } } peers { permanent_uuid: "c88b3807d6bd413bb876555a253b8680" member_type: NON_VOTER last_known_addr { host: "127.21.12.188" port: 35289 } attrs { promote: true } } } pending_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2589a7a56e404deda1060eb408ebf54c" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 41635 } } peers { permanent_uuid: "124d842e9e484f8f85ddbc038dc51a31" member_type: NON_VOTER last_known_addr { host: "127.21.12.189" port: 35823 } attrs { promote: true } } peers { permanent_uuid: "c88b3807d6bd413bb876555a253b8680" member_type: VOTER last_known_addr { host: "127.21.12.188" port: 35289 } attrs { promote: false } } }
I20260806 06:26:49.210587 31515 sys_catalog.cc:458] T 00000000000000000000000000000000 P c88b3807d6bd413bb876555a253b8680 [sys.catalog]: This master's current role is: FOLLOWER
W20260806 06:26:49.231798 31404 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 06:26:49.231998 31404 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 06:26:49.232020 31404 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 06:26:49.233151 31404 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 06:26:49.233191 31404 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 06:26:49.233214 31404 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.12.129
I20260806 06:26:49.234897 31404 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.148:39091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785997581049705-21554-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785997581049705-21554-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785997581049705-21554-0/krb5kdc/oryx_127.21.12.129.keytab
--principal=oryx/127.21.12.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.12.129:41895
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785997581049705-21554-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.12.129
--webserver_port=35087
--tserver_master_addrs=127.21.12.190:41635,127.21.12.189:35823,127.21.12.188:35289
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31404
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.12.129
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785997581049705-21554-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 05:11:02 UTC on ccycloud.quasar-hvzbbz.root.comops.site
build id 4569617
I20260806 06:26:49.235122 31404 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 06:26:49.235353 31404 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 06:26:49.241542 31545 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:26:49.241545 31540 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:26:49.241797 31539 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:26:49.241812 31404 server_base.cc:1034] running on GCE node
I20260806 06:26:49.242013 31404 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 06:26:49.242246 31404 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 06:26:49.243408 31404 hybrid_clock.cc:648] HybridClock initialized: now 1785997609243395 us; error 29 us; skew 500 ppm
Aug 06 06:26:49 dist-test-slave-4tdj krb5kdc[30146](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997609, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.12.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 06:26:49.246083 31404 init.cc:377] Logged in from keytab as oryx/127.21.12.129@KRBTEST.COM (short username oryx)
I20260806 06:26:49.246622 31404 webserver.cc:460] Webserver started at http://127.21.12.129:35087/ using document root <none> and password file <none>
I20260806 06:26:49.246831 31404 fs_manager.cc:359] Metadata directory not provided
I20260806 06:26:49.246888 31404 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 06:26:49.248210 31404 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260806 06:26:49.248765 31555 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 06:26:49.248919 31404 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260806 06:26:49.249006 31404 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785997581049705-21554-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785997581049705-21554-0/minicluster-data/ts-0/wal
uuid: "9c272304769843f1af8f68c3d16ec185"
format_stamp: "Formatted at 2026-08-06 06:26:48 on dist-test-slave-4tdj"
server_key: "2fbaa09ac62a1ae18eb3b639f348d10f"
server_key_iv: "f116fe7437f0915b44ae77dfa9d5882b"
server_key_version: "encryptionkey@0"
I20260806 06:26:49.249310 31404 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785997581049705-21554-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785997581049705-21554-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785997581049705-21554-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 06:26:49.266314 31404 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 06:26:49.268186 31404 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 06:26:49.268322 31404 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 06:26:49.268612 31404 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 06:26:49.268905 31404 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 06:26:49.268939 31404 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 06:26:49.268966 31404 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 06:26:49.268983 31404 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 06:26:49.279084 31404 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.129:41895
I20260806 06:26:49.279213 31753 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.129:41895 every 8 connection(s)
I20260806 06:26:49.279477 31404 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785997581049705-21554-0/minicluster-data/ts-0/data/info.pb
I20260806 06:26:49.286192 21554 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHhLhT/build/release/bin/kudu as pid 31404
I20260806 06:26:49.286386 21554 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHhLhT/build/release/bin/kudu with pid 30529
Aug 06 06:26:49 dist-test-slave-4tdj krb5kdc[30146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997609, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.12.129@KRBTEST.COM for oryx/127.21.12.190@KRBTEST.COM
Aug 06 06:26:49 dist-test-slave-4tdj krb5kdc[30146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997609, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.12.129@KRBTEST.COM for oryx/127.21.12.189@KRBTEST.COM
Aug 06 06:26:49 dist-test-slave-4tdj krb5kdc[30146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997609, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.12.129@KRBTEST.COM for oryx/127.21.12.188@KRBTEST.COM
I20260806 06:26:49.291685 31239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:49.280416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.129:59989 (local address 127.21.12.189:35823)
0806 06:26:49.280458 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0806 06:26:49.280460 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:26:49.280794 (+ 334us) server_negotiation.cc:407] Connection header received
0806 06:26:49.281301 (+ 507us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:26:49.281302 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:26:49.281314 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:26:49.281362 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:26:49.282473 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:49.283287 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:49.285382 (+ 2095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:49.285671 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:49.285733 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:49.288674 (+ 2941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:26:49.288676 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:26:49.288677 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 06:26:49.288685 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:26:49.290229 (+ 1544us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:26:49.290665 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:26:49.290666 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:26:49.290666 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:26:49.290725 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:26:49.290998 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:26:49.290999 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:26:49.291000 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:26:49.291136 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:26:49.291190 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0806 06:26:49.291514 (+ 324us) server_negotiation.cc:299] Negotiation successful
0806 06:26:49.291595 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260806 06:26:49.292060 31760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:49.280307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.189:35823 (local address 127.21.12.129:59989)
0806 06:26:49.280725 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0806 06:26:49.280737 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 06:26:49.281198 (+ 461us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:26:49.281435 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:26:49.281439 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:26:49.281971 (+ 532us) client_negotiation.cc:189] Negotiated authn=SASL
0806 06:26:49.282390 (+ 419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:49.282395 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:49.283444 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:49.283445 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:49.284845 (+ 1400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:49.284848 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:49.285790 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:49.285792 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:49.285874 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:49.286370 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 06:26:49.286374 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0806 06:26:49.288584 (+ 2210us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 06:26:49.290336 (+ 1752us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:26:49.290344 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:26:49.290347 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:26:49.290597 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:26:49.290794 (+ 197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:26:49.290795 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:26:49.290795 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:26:49.290921 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:26:49.291226 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 06:26:49.291234 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 06:26:49.291368 (+ 134us) client_negotiation.cc:769] Sending connection context
0806 06:26:49.291461 (+ 93us) client_negotiation.cc:240] Negotiation successful
0806 06:26:49.291618 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":401,"mutex_wait_us":35,"thread_start_us":149,"threads_started":1}
I20260806 06:26:49.292605 30277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:49.280605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.129:43365 (local address 127.21.12.190:41635)
0806 06:26:49.280682 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0806 06:26:49.280684 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:26:49.280817 (+ 133us) server_negotiation.cc:407] Connection header received
0806 06:26:49.281310 (+ 493us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:26:49.281311 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:26:49.281323 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:26:49.281353 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:26:49.283598 (+ 2245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:49.284135 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:49.284917 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:49.285222 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:49.285285 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:49.288738 (+ 3453us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:26:49.288740 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:26:49.288740 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 06:26:49.288750 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:26:49.291179 (+ 2429us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:26:49.291632 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:26:49.291633 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:26:49.291633 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:26:49.291700 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:26:49.291969 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:26:49.291970 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:26:49.291971 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:26:49.292141 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:26:49.292193 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0806 06:26:49.292423 (+ 230us) server_negotiation.cc:299] Negotiation successful
0806 06:26:49.292502 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260806 06:26:49.292613 31762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:49.280489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:41635 (local address 127.21.12.129:43365)
0806 06:26:49.280762 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0806 06:26:49.280771 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0806 06:26:49.281209 (+ 438us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:26:49.281436 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:26:49.281439 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:26:49.281931 (+ 492us) client_negotiation.cc:189] Negotiated authn=SASL
0806 06:26:49.282498 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:49.282500 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:49.284211 (+ 1711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:49.284212 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:49.284842 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:49.284843 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:49.285294 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:49.285294 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:49.285481 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:49.286045 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 06:26:49.286056 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0806 06:26:49.288630 (+ 2574us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 06:26:49.291283 (+ 2653us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:26:49.291284 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:26:49.291285 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:26:49.291529 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:26:49.291787 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:26:49.291788 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:26:49.291788 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:26:49.291876 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:26:49.292249 (+ 373us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 06:26:49.292250 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 06:26:49.292302 (+ 52us) client_negotiation.cc:769] Sending connection context
0806 06:26:49.292387 (+ 85us) client_negotiation.cc:240] Negotiation successful
0806 06:26:49.292494 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"mutex_wait_us":104,"thread_start_us":136,"threads_started":1}
I20260806 06:26:49.293157 31756 heartbeater.cc:347] Connected to a master server at 127.21.12.190:41635
I20260806 06:26:49.293267 31756 heartbeater.cc:464] Registering TS with master...
I20260806 06:26:49.293466 31756 heartbeater.cc:511] Master 127.21.12.190:41635 requested a full tablet report, sending...
I20260806 06:26:49.293987 30217 ts_manager.cc:194] Re-registered known tserver with Master: 9c272304769843f1af8f68c3d16ec185 (127.21.12.129:41895)
I20260806 06:26:49.294639 30217 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.21.12.129@KRBTEST.COM'} at 127.21.12.129:43365
I20260806 06:26:49.295878 31758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:49.280100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.129:35823 (local address 127.21.12.188:35289)
0806 06:26:49.280232 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0806 06:26:49.280234 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:26:49.280761 (+ 527us) server_negotiation.cc:407] Connection header received
0806 06:26:49.281310 (+ 549us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:26:49.281311 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:26:49.281323 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:26:49.281408 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:26:49.282563 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:49.283514 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:49.284963 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:49.285275 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:49.285335 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:49.290861 (+ 5526us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:26:49.290862 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:26:49.290863 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 06:26:49.290870 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:26:49.293753 (+ 2883us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:26:49.295106 (+ 1353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:26:49.295107 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:26:49.295108 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:26:49.295173 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:26:49.295406 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:26:49.295407 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:26:49.295408 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:26:49.295522 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:26:49.295569 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0806 06:26:49.295750 (+ 181us) server_negotiation.cc:299] Negotiation successful
0806 06:26:49.295814 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":63,"threads_started":1}
I20260806 06:26:49.296052 31761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:49.280091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.188:35289 (local address 127.21.12.129:35823)
0806 06:26:49.280605 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0806 06:26:49.280652 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0806 06:26:49.281209 (+ 557us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:26:49.281475 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:26:49.281475 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:26:49.281948 (+ 473us) client_negotiation.cc:189] Negotiated authn=SASL
0806 06:26:49.282482 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:49.282486 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:49.283585 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:49.283586 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:49.284863 (+ 1277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:49.284864 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:49.285337 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:49.285338 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:49.285512 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:49.286203 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 06:26:49.286208 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0806 06:26:49.290769 (+ 4561us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 06:26:49.293822 (+ 3053us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:26:49.293823 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:26:49.293824 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:26:49.295025 (+ 1201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:26:49.295251 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:26:49.295252 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:26:49.295252 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:26:49.295330 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:26:49.295587 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 06:26:49.295588 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 06:26:49.295643 (+ 55us) client_negotiation.cc:769] Sending connection context
0806 06:26:49.295730 (+ 87us) client_negotiation.cc:240] Negotiation successful
0806 06:26:49.295854 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":453,"mutex_wait_us":4,"thread_start_us":204,"threads_started":1}
I20260806 06:26:49.296684 31754 heartbeater.cc:347] Connected to a master server at 127.21.12.188:35289
I20260806 06:26:49.296752 31754 heartbeater.cc:464] Registering TS with master...
I20260806 06:26:49.296841 31754 heartbeater.cc:511] Master 127.21.12.188:35289 requested a full tablet report, sending...
I20260806 06:26:49.297206 31452 ts_manager.cc:194] Registered new tserver with Master: 9c272304769843f1af8f68c3d16ec185 (127.21.12.129:41895)
I20260806 06:26:49.306880 31757 heartbeater.cc:347] Connected to a master server at 127.21.12.189:35823
I20260806 06:26:49.306946 31757 heartbeater.cc:464] Registering TS with master...
I20260806 06:26:49.306995 31757 heartbeater.cc:511] Master 127.21.12.189:35823 requested a full tablet report, sending...
I20260806 06:26:49.307371 31168 ts_manager.cc:194] Registered new tserver with Master: 9c272304769843f1af8f68c3d16ec185 (127.21.12.129:41895)
I20260806 06:26:49.310384 21554 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHhLhT/build/release/bin/kudu
/tmp/dist-test-taskFHhLhT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785997581049705-21554-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785997581049705-21554-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785997581049705-21554-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785997581049705-21554-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.12.130:45263
--local_ip_for_outbound_sockets=127.21.12.130
--tserver_master_addrs=127.21.12.190:41635
--webserver_port=41827
--webserver_interface=127.21.12.130
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785997581049705-21554-0/krb5kdc/oryx_127.21.12.130.keytab
--principal=oryx/127.21.12.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.148:39091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.21.12.190:41635,127.21.12.189:35823
--tserver_master_addrs=127.21.12.190:41635,127.21.12.189:35823,127.21.12.188:35289 with env {KRB5CCNAME=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785997581049705-21554-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785997581049705-21554-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785997581049705-21554-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 06:26:49.391872 31764 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 06:26:49.392055 31764 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 06:26:49.392076 31764 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 06:26:49.393146 31764 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 06:26:49.393178 31764 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 06:26:49.393201 31764 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.12.130
I20260806 06:26:49.394649 31764 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.148:39091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785997581049705-21554-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785997581049705-21554-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785997581049705-21554-0/krb5kdc/oryx_127.21.12.130.keytab
--principal=oryx/127.21.12.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.12.130:45263
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785997581049705-21554-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.12.130
--webserver_port=41827
--tserver_master_addrs=127.21.12.190:41635,127.21.12.189:35823,127.21.12.188:35289
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31764
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.12.130
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785997581049705-21554-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 05:11:02 UTC on ccycloud.quasar-hvzbbz.root.comops.site
build id 4569617
I20260806 06:26:49.394843 31764 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 06:26:49.395021 31764 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 06:26:49.399688 31772 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:26:49.399798 31764 server_base.cc:1034] running on GCE node
W20260806 06:26:49.399744 31771 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:26:49.399755 31774 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:26:49.400077 31764 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 06:26:49.400292 31764 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 06:26:49.401429 31764 hybrid_clock.cc:648] HybridClock initialized: now 1785997609401418 us; error 28 us; skew 500 ppm
Aug 06 06:26:49 dist-test-slave-4tdj krb5kdc[30146](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997609, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.12.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 06:26:49.403791 31764 init.cc:377] Logged in from keytab as oryx/127.21.12.130@KRBTEST.COM (short username oryx)
I20260806 06:26:49.404197 31764 webserver.cc:460] Webserver started at http://127.21.12.130:41827/ using document root <none> and password file <none>
I20260806 06:26:49.404398 31764 fs_manager.cc:359] Metadata directory not provided
I20260806 06:26:49.404460 31764 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 06:26:49.405889 31764 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260806 06:26:49.406477 31784 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 06:26:49.406641 31764 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260806 06:26:49.406733 31764 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785997581049705-21554-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785997581049705-21554-0/minicluster-data/ts-1/wal
uuid: "08f0835258af4cfc8d2b541884f9ceba"
format_stamp: "Formatted at 2026-08-06 06:26:48 on dist-test-slave-4tdj"
server_key: "3794ee1678c56777f2a7ab621251d0d1"
server_key_iv: "239ef81d6d4e49e60f52bdbb4c5177d7"
server_key_version: "encryptionkey@0"
I20260806 06:26:49.407035 31764 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785997581049705-21554-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785997581049705-21554-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785997581049705-21554-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 06:26:49.421619 31764 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 06:26:49.423398 31764 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 06:26:49.423524 31764 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 06:26:49.423738 31764 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 06:26:49.424026 31764 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 06:26:49.424062 31764 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 06:26:49.424088 31764 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 06:26:49.424124 31764 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 06:26:49.429639 31764 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.130:45263
I20260806 06:26:49.429698 31947 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.130:45263 every 8 connection(s)
I20260806 06:26:49.430024 31764 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785997581049705-21554-0/minicluster-data/ts-1/data/info.pb
I20260806 06:26:49.435453 21554 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHhLhT/build/release/bin/kudu as pid 31764
I20260806 06:26:49.435638 21554 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHhLhT/build/release/bin/kudu with pid 30731
Aug 06 06:26:49 dist-test-slave-4tdj krb5kdc[30146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997609, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.12.130@KRBTEST.COM for oryx/127.21.12.189@KRBTEST.COM
Aug 06 06:26:49 dist-test-slave-4tdj krb5kdc[30146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997609, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.12.130@KRBTEST.COM for oryx/127.21.12.188@KRBTEST.COM
Aug 06 06:26:49 dist-test-slave-4tdj krb5kdc[30146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997609, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.12.130@KRBTEST.COM for oryx/127.21.12.190@KRBTEST.COM
I20260806 06:26:49.441813 31758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:49.430747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.130:44853 (local address 127.21.12.188:35289)
0806 06:26:49.430830 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0806 06:26:49.430832 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:26:49.431073 (+ 241us) server_negotiation.cc:407] Connection header received
0806 06:26:49.432152 (+ 1079us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:26:49.432154 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:26:49.432169 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:26:49.432200 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:26:49.432901 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:49.433665 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:49.435157 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:49.435477 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:49.435531 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:49.438693 (+ 3162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:26:49.438696 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:26:49.438696 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 06:26:49.438703 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:26:49.440263 (+ 1560us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:26:49.440737 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:26:49.440739 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:26:49.440739 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:26:49.440797 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:26:49.441083 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:26:49.441084 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:26:49.441084 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:26:49.441219 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:26:49.441259 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0806 06:26:49.441628 (+ 369us) server_negotiation.cc:299] Negotiation successful
0806 06:26:49.441719 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260806 06:26:49.442171 31239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:49.430746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.130:56617 (local address 127.21.12.189:35823)
0806 06:26:49.430795 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0806 06:26:49.430797 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:26:49.431074 (+ 277us) server_negotiation.cc:407] Connection header received
0806 06:26:49.432156 (+ 1082us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:26:49.432157 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:26:49.432169 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:26:49.432200 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:26:49.433768 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:49.434262 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:49.435099 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:49.435401 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:49.435459 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:49.439366 (+ 3907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:26:49.439368 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:26:49.439369 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 06:26:49.439377 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:26:49.441083 (+ 1706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:26:49.441414 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:26:49.441415 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:26:49.441416 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:26:49.441468 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:26:49.441669 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:26:49.441670 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:26:49.441670 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:26:49.441790 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:26:49.441830 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0806 06:26:49.442018 (+ 188us) server_negotiation.cc:299] Negotiation successful
0806 06:26:49.442085 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260806 06:26:49.442319 31953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:49.430761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.188:35289 (local address 127.21.12.130:44853)
0806 06:26:49.430933 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0806 06:26:49.430981 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0806 06:26:49.432023 (+ 1042us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:26:49.432264 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:26:49.432265 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:26:49.432590 (+ 325us) client_negotiation.cc:189] Negotiated authn=SASL
0806 06:26:49.432778 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:49.432784 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:49.433777 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:49.433779 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:49.435068 (+ 1289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:49.435071 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:49.435666 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:49.435667 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:49.435952 (+ 285us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:49.436304 (+ 352us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 06:26:49.436309 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0806 06:26:49.438583 (+ 2274us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 06:26:49.440347 (+ 1764us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:26:49.440356 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:26:49.440358 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:26:49.440646 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:26:49.440868 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:26:49.440869 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:26:49.440869 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:26:49.440995 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:26:49.441305 (+ 310us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 06:26:49.441312 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 06:26:49.441467 (+ 155us) client_negotiation.cc:769] Sending connection context
0806 06:26:49.441904 (+ 437us) client_negotiation.cc:240] Negotiation successful
0806 06:26:49.442087 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"mutex_wait_us":46,"thread_start_us":95,"threads_started":1}
I20260806 06:26:49.442474 31955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:49.430761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.189:35823 (local address 127.21.12.130:56617)
0806 06:26:49.430933 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0806 06:26:49.430981 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0806 06:26:49.432030 (+ 1049us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:26:49.432264 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:26:49.432265 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:26:49.432598 (+ 333us) client_negotiation.cc:189] Negotiated authn=SASL
0806 06:26:49.432778 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:49.432784 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:49.434381 (+ 1597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:49.434382 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:49.435021 (+ 639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:49.435023 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:49.435525 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:49.435526 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:49.435669 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:49.436396 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 06:26:49.436400 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0806 06:26:49.439266 (+ 2866us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 06:26:49.441161 (+ 1895us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:26:49.441161 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:26:49.441162 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:26:49.441342 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:26:49.441534 (+ 192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:26:49.441535 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:26:49.441535 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:26:49.441602 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:26:49.441876 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 06:26:49.441877 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 06:26:49.441914 (+ 37us) client_negotiation.cc:769] Sending connection context
0806 06:26:49.442243 (+ 329us) client_negotiation.cc:240] Negotiation successful
0806 06:26:49.442352 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"mutex_wait_us":7,"thread_start_us":90,"threads_started":1}
I20260806 06:26:49.447508 31951 heartbeater.cc:347] Connected to a master server at 127.21.12.189:35823
I20260806 06:26:49.447615 31951 heartbeater.cc:464] Registering TS with master...
I20260806 06:26:49.447798 31951 heartbeater.cc:511] Master 127.21.12.189:35823 requested a full tablet report, sending...
I20260806 06:26:49.448303 31168 ts_manager.cc:194] Registered new tserver with Master: 08f0835258af4cfc8d2b541884f9ceba (127.21.12.130:45263)
I20260806 06:26:49.451259 30277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:49.431154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.130:57771 (local address 127.21.12.190:41635)
0806 06:26:49.431201 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0806 06:26:49.431203 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:26:49.431250 (+ 47us) server_negotiation.cc:407] Connection header received
0806 06:26:49.432112 (+ 862us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:26:49.432113 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:26:49.432125 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:26:49.432180 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:26:49.432864 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:49.433674 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:49.435206 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:49.435477 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:49.435545 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:49.448356 (+ 12811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:26:49.448358 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:26:49.448359 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 06:26:49.448369 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:26:49.450158 (+ 1789us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:26:49.450541 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:26:49.450542 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:26:49.450543 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:26:49.450600 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:26:49.450771 (+ 171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:26:49.450772 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:26:49.450773 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:26:49.450891 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:26:49.450955 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0806 06:26:49.451100 (+ 145us) server_negotiation.cc:299] Negotiation successful
0806 06:26:49.451176 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260806 06:26:49.451355 31956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:49.431048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:41635 (local address 127.21.12.130:57771)
0806 06:26:49.431174 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0806 06:26:49.431188 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 06:26:49.432028 (+ 840us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:26:49.432248 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:26:49.432252 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:26:49.432576 (+ 324us) client_negotiation.cc:189] Negotiated authn=SASL
0806 06:26:49.432778 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:49.432782 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:49.433777 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:49.433779 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:49.435148 (+ 1369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:49.435150 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:49.435701 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:49.435701 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:49.435760 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:49.436221 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 06:26:49.436233 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 06:26:49.448253 (+ 12020us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 06:26:49.450243 (+ 1990us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:26:49.450244 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:26:49.450245 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:26:49.450467 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:26:49.450656 (+ 189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:26:49.450656 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:26:49.450657 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:26:49.450713 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:26:49.450945 (+ 232us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 06:26:49.450946 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 06:26:49.450988 (+ 42us) client_negotiation.cc:769] Sending connection context
0806 06:26:49.451043 (+ 55us) client_negotiation.cc:240] Negotiation successful
0806 06:26:49.451146 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"mutex_wait_us":27,"thread_start_us":79,"threads_started":1}
I20260806 06:26:49.451666 31949 heartbeater.cc:347] Connected to a master server at 127.21.12.188:35289
I20260806 06:26:49.451715 31949 heartbeater.cc:464] Registering TS with master...
I20260806 06:26:49.451767 31949 heartbeater.cc:511] Master 127.21.12.188:35289 requested a full tablet report, sending...
I20260806 06:26:49.452149 31452 ts_manager.cc:194] Registered new tserver with Master: 08f0835258af4cfc8d2b541884f9ceba (127.21.12.130:45263)
I20260806 06:26:49.452417 31950 heartbeater.cc:347] Connected to a master server at 127.21.12.190:41635
I20260806 06:26:49.452467 31950 heartbeater.cc:464] Registering TS with master...
I20260806 06:26:49.452531 31950 heartbeater.cc:511] Master 127.21.12.190:41635 requested a full tablet report, sending...
I20260806 06:26:49.452744 30217 ts_manager.cc:194] Re-registered known tserver with Master: 08f0835258af4cfc8d2b541884f9ceba (127.21.12.130:45263)
I20260806 06:26:49.453172 30217 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.21.12.130@KRBTEST.COM'} at 127.21.12.130:57771
I20260806 06:26:49.456542 21554 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHhLhT/build/release/bin/kudu
/tmp/dist-test-taskFHhLhT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785997581049705-21554-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785997581049705-21554-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785997581049705-21554-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785997581049705-21554-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.12.131:46131
--local_ip_for_outbound_sockets=127.21.12.131
--tserver_master_addrs=127.21.12.190:41635
--webserver_port=43011
--webserver_interface=127.21.12.131
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785997581049705-21554-0/krb5kdc/oryx_127.21.12.131.keytab
--principal=oryx/127.21.12.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.148:39091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.21.12.190:41635,127.21.12.189:35823
--tserver_master_addrs=127.21.12.190:41635,127.21.12.189:35823,127.21.12.188:35289 with env {KRB5CCNAME=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785997581049705-21554-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785997581049705-21554-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785997581049705-21554-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 06:26:49.522426 31106 raft_consensus.cc:1013] T 00000000000000000000000000000000 P 2589a7a56e404deda1060eb408ebf54c [term 1 LEADER]: attempt to promote peer 124d842e9e484f8f85ddbc038dc51a31: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20260806 06:26:49.537322 31957 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 06:26:49.537508 31957 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 06:26:49.537537 31957 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 06:26:49.538697 31957 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 06:26:49.538735 31957 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 06:26:49.538769 31957 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.12.131
I20260806 06:26:49.540220 31957 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.148:39091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785997581049705-21554-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785997581049705-21554-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785997581049705-21554-0/krb5kdc/oryx_127.21.12.131.keytab
--principal=oryx/127.21.12.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.12.131:46131
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785997581049705-21554-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.12.131
--webserver_port=43011
--tserver_master_addrs=127.21.12.190:41635,127.21.12.189:35823,127.21.12.188:35289
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31957
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.12.131
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785997581049705-21554-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 05:11:02 UTC on ccycloud.quasar-hvzbbz.root.comops.site
build id 4569617
I20260806 06:26:49.540442 31957 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 06:26:49.540650 31957 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 06:26:49.546326 31964 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:26:49.546319 31963 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:26:49.546319 31966 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:26:49.546345 31957 server_base.cc:1034] running on GCE node
I20260806 06:26:49.546666 31957 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 06:26:49.546861 31957 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 06:26:49.547991 31957 hybrid_clock.cc:648] HybridClock initialized: now 1785997609547977 us; error 31 us; skew 500 ppm
Aug 06 06:26:49 dist-test-slave-4tdj krb5kdc[30146](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997609, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.12.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 06:26:49.550408 31957 init.cc:377] Logged in from keytab as oryx/127.21.12.131@KRBTEST.COM (short username oryx)
I20260806 06:26:49.550788 31957 webserver.cc:460] Webserver started at http://127.21.12.131:43011/ using document root <none> and password file <none>
I20260806 06:26:49.550977 31957 fs_manager.cc:359] Metadata directory not provided
I20260806 06:26:49.551029 31957 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 06:26:49.552471 31957 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260806 06:26:49.553140 31977 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 06:26:49.553318 31957 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 06:26:49.553406 31957 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785997581049705-21554-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785997581049705-21554-0/minicluster-data/ts-2/wal
uuid: "79d4ece6336b48e59c3e65c1d11c3ded"
format_stamp: "Formatted at 2026-08-06 06:26:48 on dist-test-slave-4tdj"
server_key: "82c0cde3ca11677e7935a3dee2fca78d"
server_key_iv: "63b32f56a89f84661cf91ad3c7298acd"
server_key_version: "encryptionkey@0"
I20260806 06:26:49.553717 31957 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785997581049705-21554-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785997581049705-21554-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785997581049705-21554-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 06:26:49.564070 31957 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 06:26:49.565663 31957 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 06:26:49.565768 31957 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 06:26:49.565932 31957 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 06:26:49.566175 31957 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 06:26:49.566205 31957 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 06:26:49.566223 31957 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 06:26:49.566239 31957 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 06:26:49.572088 31957 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.131:46131
I20260806 06:26:49.572151 32151 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.131:46131 every 8 connection(s)
I20260806 06:26:49.572440 31957 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785997581049705-21554-0/minicluster-data/ts-2/data/info.pb
Aug 06 06:26:49 dist-test-slave-4tdj krb5kdc[30146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997609, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.12.131@KRBTEST.COM for oryx/127.21.12.190@KRBTEST.COM
Aug 06 06:26:49 dist-test-slave-4tdj krb5kdc[30146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997609, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.12.131@KRBTEST.COM for oryx/127.21.12.188@KRBTEST.COM
Aug 06 06:26:49 dist-test-slave-4tdj krb5kdc[30146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997609, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.12.131@KRBTEST.COM for oryx/127.21.12.189@KRBTEST.COM
I20260806 06:26:49.581651 21554 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHhLhT/build/release/bin/kudu as pid 31957
I20260806 06:26:49.583458 30277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:49.573687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.131:33059 (local address 127.21.12.190:41635)
0806 06:26:49.573748 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0806 06:26:49.573750 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:26:49.573947 (+ 197us) server_negotiation.cc:407] Connection header received
0806 06:26:49.574763 (+ 816us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:26:49.574765 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:26:49.574777 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:26:49.574806 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:26:49.575596 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:49.576357 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:49.577515 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:49.577830 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:49.577890 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:49.580162 (+ 2272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:26:49.580164 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:26:49.580164 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 06:26:49.580170 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:26:49.581848 (+ 1678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:26:49.582312 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:26:49.582313 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:26:49.582314 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:26:49.582378 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:26:49.582663 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:26:49.582663 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:26:49.582664 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:26:49.582810 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:26:49.582855 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0806 06:26:49.583261 (+ 406us) server_negotiation.cc:299] Negotiation successful
0806 06:26:49.583344 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260806 06:26:49.583901 31758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:49.573136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.131:47139 (local address 127.21.12.188:35289)
0806 06:26:49.573187 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0806 06:26:49.573189 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:26:49.573624 (+ 435us) server_negotiation.cc:407] Connection header received
0806 06:26:49.574865 (+ 1241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:26:49.574867 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:26:49.574877 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:26:49.574931 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:26:49.575590 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:49.576363 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:49.577477 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:49.577680 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:49.577736 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:49.580567 (+ 2831us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:26:49.580568 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:26:49.580568 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 06:26:49.580576 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:26:49.582130 (+ 1554us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:26:49.582487 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:26:49.582489 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:26:49.582489 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:26:49.582564 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:26:49.583314 (+ 750us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:26:49.583315 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:26:49.583315 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:26:49.583464 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:26:49.583517 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0806 06:26:49.583738 (+ 221us) server_negotiation.cc:299] Negotiation successful
0806 06:26:49.583814 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260806 06:26:49.584064 32165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:49.573608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:41635 (local address 127.21.12.131:33059)
0806 06:26:49.573877 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0806 06:26:49.573885 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0806 06:26:49.574625 (+ 740us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:26:49.574930 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:26:49.574931 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:26:49.575283 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0806 06:26:49.575469 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:49.575474 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:49.576469 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:49.576470 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:49.577408 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:49.577410 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:49.577942 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:49.577943 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:49.578001 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:49.578515 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 06:26:49.578520 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0806 06:26:49.580080 (+ 1560us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 06:26:49.581932 (+ 1852us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:26:49.581938 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:26:49.581940 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:26:49.582201 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:26:49.582456 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:26:49.582456 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:26:49.582456 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:26:49.582579 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:26:49.582912 (+ 333us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 06:26:49.582919 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 06:26:49.583068 (+ 149us) client_negotiation.cc:769] Sending connection context
0806 06:26:49.583583 (+ 515us) client_negotiation.cc:240] Negotiation successful
0806 06:26:49.583750 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"mutex_wait_us":29,"thread_start_us":111,"threads_started":1}
I20260806 06:26:49.584470 32161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:49.573143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.188:35289 (local address 127.21.12.131:47139)
0806 06:26:49.573463 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0806 06:26:49.573531 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0806 06:26:49.574626 (+ 1095us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:26:49.574946 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:26:49.574947 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:26:49.575268 (+ 321us) client_negotiation.cc:189] Negotiated authn=SASL
0806 06:26:49.575470 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:49.575476 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:49.576462 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:49.576462 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:49.577395 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:49.577396 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:49.577752 (+ 356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:49.577753 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:49.577880 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:49.578372 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 06:26:49.578381 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0806 06:26:49.580491 (+ 2110us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 06:26:49.582213 (+ 1722us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:26:49.582214 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:26:49.582214 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:26:49.582418 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:26:49.583152 (+ 734us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:26:49.583153 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:26:49.583153 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:26:49.583222 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:26:49.583577 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 06:26:49.583578 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 06:26:49.583620 (+ 42us) client_negotiation.cc:769] Sending connection context
0806 06:26:49.584053 (+ 433us) client_negotiation.cc:240] Negotiation successful
0806 06:26:49.584369 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"mutex_wait_us":54,"thread_start_us":216,"threads_started":1}
I20260806 06:26:49.584539 32154 heartbeater.cc:347] Connected to a master server at 127.21.12.190:41635
I20260806 06:26:49.584630 32154 heartbeater.cc:464] Registering TS with master...
I20260806 06:26:49.584805 32154 heartbeater.cc:511] Master 127.21.12.190:41635 requested a full tablet report, sending...
I20260806 06:26:49.585088 32157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:49.573143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.189:35823 (local address 127.21.12.131:59131)
0806 06:26:49.573515 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0806 06:26:49.573530 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 06:26:49.574619 (+ 1089us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:26:49.574877 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:26:49.574880 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:26:49.575268 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0806 06:26:49.575470 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:49.575476 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:49.577272 (+ 1796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:49.577273 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:49.578014 (+ 741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:49.578015 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:49.578465 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:49.578466 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:49.578529 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:49.578895 (+ 366us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 06:26:49.578899 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0806 06:26:49.580944 (+ 2045us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 06:26:49.583820 (+ 2876us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:26:49.583821 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:26:49.583821 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:26:49.584006 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:26:49.584235 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:26:49.584236 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:26:49.584236 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:26:49.584293 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:26:49.584626 (+ 333us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 06:26:49.584627 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 06:26:49.584668 (+ 41us) client_negotiation.cc:769] Sending connection context
0806 06:26:49.584729 (+ 61us) client_negotiation.cc:240] Negotiation successful
0806 06:26:49.584816 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":348,"mutex_wait_us":33,"spinlock_wait_cycles":6912,"thread_start_us":202,"threads_started":1}
I20260806 06:26:49.585404 31239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:49.573135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.131:59131 (local address 127.21.12.189:35823)
0806 06:26:49.573221 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0806 06:26:49.573223 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:26:49.573626 (+ 403us) server_negotiation.cc:407] Connection header received
0806 06:26:49.574742 (+ 1116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:26:49.574744 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:26:49.574757 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:26:49.574807 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:26:49.576451 (+ 1644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:49.577188 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:49.578081 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:49.578399 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:49.578463 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:49.581054 (+ 2591us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:26:49.581056 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:26:49.581056 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 06:26:49.581065 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:26:49.583705 (+ 2640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:26:49.584103 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:26:49.584105 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:26:49.584105 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:26:49.584164 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:26:49.584364 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:26:49.584366 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:26:49.584366 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:26:49.584516 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:26:49.584582 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0806 06:26:49.585241 (+ 659us) server_negotiation.cc:299] Negotiation successful
0806 06:26:49.585321 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260806 06:26:49.585582 30217 ts_manager.cc:194] Re-registered known tserver with Master: 79d4ece6336b48e59c3e65c1d11c3ded (127.21.12.131:46131)
I20260806 06:26:49.585731 32155 heartbeater.cc:347] Connected to a master server at 127.21.12.189:35823
I20260806 06:26:49.585773 32155 heartbeater.cc:464] Registering TS with master...
I20260806 06:26:49.585829 32155 heartbeater.cc:511] Master 127.21.12.189:35823 requested a full tablet report, sending...
I20260806 06:26:49.586006 32153 heartbeater.cc:347] Connected to a master server at 127.21.12.188:35289
I20260806 06:26:49.586047 32153 heartbeater.cc:464] Registering TS with master...
I20260806 06:26:49.586051 30217 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.21.12.131@KRBTEST.COM'} at 127.21.12.131:33059
I20260806 06:26:49.586099 32153 heartbeater.cc:511] Master 127.21.12.188:35289 requested a full tablet report, sending...
I20260806 06:26:49.586131 31168 ts_manager.cc:194] Registered new tserver with Master: 79d4ece6336b48e59c3e65c1d11c3ded (127.21.12.131:46131)
I20260806 06:26:49.586534 31452 ts_manager.cc:194] Registered new tserver with Master: 79d4ece6336b48e59c3e65c1d11c3ded (127.21.12.131:46131)
I20260806 06:26:49.591816 32172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:49.582921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59486 (local address 127.21.12.189:35823)
0806 06:26:49.584584 (+ 1663us) server_negotiation.cc:206] Beginning negotiation
0806 06:26:49.584586 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:26:49.584594 (+ 8us) server_negotiation.cc:407] Connection header received
0806 06:26:49.584652 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:26:49.584653 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:26:49.584665 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:26:49.585258 (+ 593us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:26:49.585666 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:49.586339 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:49.587333 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:49.587659 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:49.588031 (+ 372us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:49.589166 (+ 1135us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:26:49.589168 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:26:49.589168 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 06:26:49.589174 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:26:49.590672 (+ 1498us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:26:49.590934 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:26:49.590934 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:26:49.590935 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:26:49.590988 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:26:49.591148 (+ 160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:26:49.591150 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:26:49.591150 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:26:49.591259 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:26:49.591345 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0806 06:26:49.591653 (+ 308us) server_negotiation.cc:299] Negotiation successful
0806 06:26:49.591726 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1643,"thread_start_us":53,"threads_started":1}
I20260806 06:26:49.592621 32171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:49.582729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50786 (local address 127.21.12.190:41635)
0806 06:26:49.582961 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0806 06:26:49.582963 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:26:49.584682 (+ 1719us) server_negotiation.cc:407] Connection header received
0806 06:26:49.585280 (+ 598us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:26:49.585281 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:26:49.585294 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:26:49.585335 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:26:49.586948 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:49.587413 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:49.588108 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:49.588380 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:49.588410 (+ 30us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:49.589742 (+ 1332us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:26:49.589744 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:26:49.589744 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 06:26:49.589751 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:26:49.591506 (+ 1755us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:26:49.591885 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:26:49.591886 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:26:49.591887 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:26:49.591952 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:26:49.592127 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:26:49.592128 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:26:49.592129 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:26:49.592266 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:26:49.592364 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0806 06:26:49.592448 (+ 84us) server_negotiation.cc:299] Negotiation successful
0806 06:26:49.592534 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":201,"thread_start_us":101,"threads_started":1}
I20260806 06:26:49.593268 31758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:49.583376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35326 (local address 127.21.12.188:35289)
0806 06:26:49.583990 (+ 614us) server_negotiation.cc:206] Beginning negotiation
0806 06:26:49.583992 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:26:49.584806 (+ 814us) server_negotiation.cc:407] Connection header received
0806 06:26:49.584899 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:26:49.584900 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:26:49.584909 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:26:49.584935 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:26:49.587287 (+ 2352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:49.587945 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:49.588949 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:49.589221 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:49.589292 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:49.590649 (+ 1357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:26:49.590650 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:26:49.590651 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 06:26:49.590659 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:26:49.592055 (+ 1396us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:26:49.592499 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:26:49.592500 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:26:49.592500 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:26:49.592552 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:26:49.592843 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:26:49.592844 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:26:49.592844 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:26:49.592946 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:26:49.593039 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0806 06:26:49.593118 (+ 79us) server_negotiation.cc:299] Negotiation successful
0806 06:26:49.593190 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":601,"thread_start_us":59,"threads_started":1}
I20260806 06:26:49.594231 30217 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50786:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 06:26:49.595902 30217 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 06:26:49.597532 31400 raft_consensus.cc:1013] T 00000000000000000000000000000000 P 2589a7a56e404deda1060eb408ebf54c [term 1 LEADER]: attempt to promote peer 124d842e9e484f8f85ddbc038dc51a31: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20260806 06:26:49.651978 31758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:49.649353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35332 (local address 127.21.12.188:35289)
0806 06:26:49.649418 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0806 06:26:49.649420 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:26:49.649523 (+ 103us) server_negotiation.cc:407] Connection header received
0806 06:26:49.649558 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:26:49.649558 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:26:49.649583 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:26:49.649651 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 06:26:49.649765 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:49.650250 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:49.651163 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:49.651682 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:49.651734 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:49.651799 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0806 06:26:49.651892 (+ 93us) server_negotiation.cc:299] Negotiation successful
0806 06:26:49.651921 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260806 06:26:49.652000 32184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:49.649259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.188:35289 (local address 127.0.0.1:35332)
0806 06:26:49.649413 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0806 06:26:49.649425 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 06:26:49.649499 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:26:49.649661 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:26:49.649661 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:26:49.649664 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 06:26:49.649720 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:49.649721 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:49.650342 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:49.650343 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:49.651082 (+ 739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:49.651085 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:49.651769 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:49.651770 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:49.651837 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:49.651841 (+ 4us) client_negotiation.cc:769] Sending connection context
0806 06:26:49.651883 (+ 42us) client_negotiation.cc:240] Negotiation successful
0806 06:26:49.651918 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":64,"threads_started":1}
I20260806 06:26:49.652434 31466 raft_consensus.cc:1203] T 00000000000000000000000000000000 P c88b3807d6bd413bb876555a253b8680 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.6->[1.7-1.8] Dedup: 1.7->[1.8-1.8]
I20260806 06:26:49.652822 31515 sys_catalog.cc:455] T 00000000000000000000000000000000 P c88b3807d6bd413bb876555a253b8680 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2589a7a56e404deda1060eb408ebf54c. Latest consensus state: current_term: 1 leader_uuid: "2589a7a56e404deda1060eb408ebf54c" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "2589a7a56e404deda1060eb408ebf54c" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 41635 } } peers { permanent_uuid: "124d842e9e484f8f85ddbc038dc51a31" member_type: NON_VOTER last_known_addr { host: "127.21.12.189" port: 35823 } attrs { promote: true } } peers { permanent_uuid: "c88b3807d6bd413bb876555a253b8680" member_type: NON_VOTER last_known_addr { host: "127.21.12.188" port: 35289 } attrs { promote: true } } } pending_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2589a7a56e404deda1060eb408ebf54c" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 41635 } } peers { permanent_uuid: "124d842e9e484f8f85ddbc038dc51a31" member_type: NON_VOTER last_known_addr { host: "127.21.12.189" port: 35823 } attrs { promote: true } } peers { permanent_uuid: "c88b3807d6bd413bb876555a253b8680" member_type: VOTER last_known_addr { host: "127.21.12.188" port: 35289 } attrs { promote: false } } }
I20260806 06:26:49.652954 31515 sys_catalog.cc:458] T 00000000000000000000000000000000 P c88b3807d6bd413bb876555a253b8680 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 06:26:49.653393 31106 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 2589a7a56e404deda1060eb408ebf54c [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, c88b3807d6bd413bb876555a253b8680 (127.21.12.188) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2589a7a56e404deda1060eb408ebf54c" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 41635 } } peers { permanent_uuid: "124d842e9e484f8f85ddbc038dc51a31" member_type: NON_VOTER last_known_addr { host: "127.21.12.189" port: 35823 } attrs { promote: true } } peers { permanent_uuid: "c88b3807d6bd413bb876555a253b8680" member_type: VOTER last_known_addr { host: "127.21.12.188" port: 35289 } attrs { promote: false } } }
I20260806 06:26:49.653568 31466 raft_consensus.cc:2932] T 00000000000000000000000000000000 P c88b3807d6bd413bb876555a253b8680 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, c88b3807d6bd413bb876555a253b8680 (127.21.12.188) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2589a7a56e404deda1060eb408ebf54c" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 41635 } } peers { permanent_uuid: "124d842e9e484f8f85ddbc038dc51a31" member_type: NON_VOTER last_known_addr { host: "127.21.12.189" port: 35823 } attrs { promote: true } } peers { permanent_uuid: "c88b3807d6bd413bb876555a253b8680" member_type: VOTER last_known_addr { host: "127.21.12.188" port: 35289 } attrs { promote: false } } }
I20260806 06:26:49.654505 31515 sys_catalog.cc:455] T 00000000000000000000000000000000 P c88b3807d6bd413bb876555a253b8680 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2589a7a56e404deda1060eb408ebf54c" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2589a7a56e404deda1060eb408ebf54c" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 41635 } } peers { permanent_uuid: "124d842e9e484f8f85ddbc038dc51a31" member_type: NON_VOTER last_known_addr { host: "127.21.12.189" port: 35823 } attrs { promote: true } } peers { permanent_uuid: "c88b3807d6bd413bb876555a253b8680" member_type: VOTER last_known_addr { host: "127.21.12.188" port: 35289 } attrs { promote: false } } }
I20260806 06:26:49.654593 31515 sys_catalog.cc:458] T 00000000000000000000000000000000 P c88b3807d6bd413bb876555a253b8680 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 06:26:49.654553 31106 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2589a7a56e404deda1060eb408ebf54c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2589a7a56e404deda1060eb408ebf54c" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2589a7a56e404deda1060eb408ebf54c" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 41635 } } peers { permanent_uuid: "124d842e9e484f8f85ddbc038dc51a31" member_type: NON_VOTER last_known_addr { host: "127.21.12.189" port: 35823 } attrs { promote: true } } peers { permanent_uuid: "c88b3807d6bd413bb876555a253b8680" member_type: VOTER last_known_addr { host: "127.21.12.188" port: 35289 } attrs { promote: false } } }
I20260806 06:26:49.654634 31106 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2589a7a56e404deda1060eb408ebf54c [sys.catalog]: This master's current role is: LEADER
I20260806 06:26:49.654770 31400 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2589a7a56e404deda1060eb408ebf54c [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "2589a7a56e404deda1060eb408ebf54c" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2589a7a56e404deda1060eb408ebf54c" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 41635 } } peers { permanent_uuid: "124d842e9e484f8f85ddbc038dc51a31" member_type: NON_VOTER last_known_addr { host: "127.21.12.189" port: 35823 } attrs { promote: true } } peers { permanent_uuid: "c88b3807d6bd413bb876555a253b8680" member_type: VOTER last_known_addr { host: "127.21.12.188" port: 35289 } attrs { promote: false } } }
I20260806 06:26:49.654837 31400 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2589a7a56e404deda1060eb408ebf54c [sys.catalog]: This master's current role is: LEADER
I20260806 06:26:49.655087 31181 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 124d842e9e484f8f85ddbc038dc51a31 [term 1 LEARNER]: Committing config change with OpId 1.7: config changed from index 6 to 7, c88b3807d6bd413bb876555a253b8680 (127.21.12.188) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2589a7a56e404deda1060eb408ebf54c" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 41635 } } peers { permanent_uuid: "124d842e9e484f8f85ddbc038dc51a31" member_type: NON_VOTER last_known_addr { host: "127.21.12.189" port: 35823 } attrs { promote: true } } peers { permanent_uuid: "c88b3807d6bd413bb876555a253b8680" member_type: VOTER last_known_addr { host: "127.21.12.188" port: 35289 } attrs { promote: false } } }
I20260806 06:26:49.656158 32196 sys_catalog.cc:455] T 00000000000000000000000000000000 P 124d842e9e484f8f85ddbc038dc51a31 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2589a7a56e404deda1060eb408ebf54c" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2589a7a56e404deda1060eb408ebf54c" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 41635 } } peers { permanent_uuid: "124d842e9e484f8f85ddbc038dc51a31" member_type: NON_VOTER last_known_addr { host: "127.21.12.189" port: 35823 } attrs { promote: true } } peers { permanent_uuid: "c88b3807d6bd413bb876555a253b8680" member_type: VOTER last_known_addr { host: "127.21.12.188" port: 35289 } attrs { promote: false } } }
I20260806 06:26:49.656255 32196 sys_catalog.cc:458] T 00000000000000000000000000000000 P 124d842e9e484f8f85ddbc038dc51a31 [sys.catalog]: This master's current role is: LEARNER
I20260806 06:26:49.656383 31106 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 2589a7a56e404deda1060eb408ebf54c: attempting to promote NON_VOTER 124d842e9e484f8f85ddbc038dc51a31 to VOTER
I20260806 06:26:49.656503 31106 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2589a7a56e404deda1060eb408ebf54c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "2589a7a56e404deda1060eb408ebf54c" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 41635 } } peers { permanent_uuid: "124d842e9e484f8f85ddbc038dc51a31" member_type: VOTER last_known_addr { host: "127.21.12.189" port: 35823 } attrs { promote: false } } peers { permanent_uuid: "c88b3807d6bd413bb876555a253b8680" member_type: VOTER last_known_addr { host: "127.21.12.188" port: 35289 } attrs { promote: false } }
I20260806 06:26:49.656863 31466 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c88b3807d6bd413bb876555a253b8680 [term 1 FOLLOWER]: Refusing update from remote peer 2589a7a56e404deda1060eb408ebf54c: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20260806 06:26:49.657078 31400 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2589a7a56e404deda1060eb408ebf54c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c88b3807d6bd413bb876555a253b8680" member_type: VOTER last_known_addr { host: "127.21.12.188" port: 35289 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.000s
I20260806 06:26:49.657116 31181 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 124d842e9e484f8f85ddbc038dc51a31 [term 1 LEARNER]: Refusing update from remote peer 2589a7a56e404deda1060eb408ebf54c: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20260806 06:26:49.657307 31400 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2589a7a56e404deda1060eb408ebf54c [LEADER]: Connected to new peer: Peer: permanent_uuid: "124d842e9e484f8f85ddbc038dc51a31" member_type: VOTER last_known_addr { host: "127.21.12.189" port: 35823 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.000s
I20260806 06:26:49.657757 32193 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 2589a7a56e404deda1060eb408ebf54c [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 7 to 10, 124d842e9e484f8f85ddbc038dc51a31 (127.21.12.189) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "2589a7a56e404deda1060eb408ebf54c" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 41635 } } peers { permanent_uuid: "124d842e9e484f8f85ddbc038dc51a31" member_type: VOTER last_known_addr { host: "127.21.12.189" port: 35823 } attrs { promote: false } } peers { permanent_uuid: "c88b3807d6bd413bb876555a253b8680" member_type: VOTER last_known_addr { host: "127.21.12.188" port: 35289 } attrs { promote: false } } }
I20260806 06:26:49.657872 31466 raft_consensus.cc:2932] T 00000000000000000000000000000000 P c88b3807d6bd413bb876555a253b8680 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 7 to 10, 124d842e9e484f8f85ddbc038dc51a31 (127.21.12.189) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "2589a7a56e404deda1060eb408ebf54c" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 41635 } } peers { permanent_uuid: "124d842e9e484f8f85ddbc038dc51a31" member_type: VOTER last_known_addr { host: "127.21.12.189" port: 35823 } attrs { promote: false } } peers { permanent_uuid: "c88b3807d6bd413bb876555a253b8680" member_type: VOTER last_known_addr { host: "127.21.12.188" port: 35289 } attrs { promote: false } } }
I20260806 06:26:49.658704 32193 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2589a7a56e404deda1060eb408ebf54c [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "2589a7a56e404deda1060eb408ebf54c" committed_config { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "2589a7a56e404deda1060eb408ebf54c" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 41635 } } peers { permanent_uuid: "124d842e9e484f8f85ddbc038dc51a31" member_type: VOTER last_known_addr { host: "127.21.12.189" port: 35823 } attrs { promote: false } } peers { permanent_uuid: "c88b3807d6bd413bb876555a253b8680" member_type: VOTER last_known_addr { host: "127.21.12.188" port: 35289 } attrs { promote: false } } }
I20260806 06:26:49.658775 32193 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2589a7a56e404deda1060eb408ebf54c [sys.catalog]: This master's current role is: LEADER
I20260806 06:26:49.658831 32193 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2589a7a56e404deda1060eb408ebf54c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2589a7a56e404deda1060eb408ebf54c" committed_config { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "2589a7a56e404deda1060eb408ebf54c" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 41635 } } peers { permanent_uuid: "124d842e9e484f8f85ddbc038dc51a31" member_type: VOTER last_known_addr { host: "127.21.12.189" port: 35823 } attrs { promote: false } } peers { permanent_uuid: "c88b3807d6bd413bb876555a253b8680" member_type: VOTER last_known_addr { host: "127.21.12.188" port: 35289 } attrs { promote: false } } }
I20260806 06:26:49.658870 32193 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2589a7a56e404deda1060eb408ebf54c [sys.catalog]: This master's current role is: LEADER
I20260806 06:26:49.658879 31515 sys_catalog.cc:455] T 00000000000000000000000000000000 P c88b3807d6bd413bb876555a253b8680 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2589a7a56e404deda1060eb408ebf54c" committed_config { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "2589a7a56e404deda1060eb408ebf54c" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 41635 } } peers { permanent_uuid: "124d842e9e484f8f85ddbc038dc51a31" member_type: VOTER last_known_addr { host: "127.21.12.189" port: 35823 } attrs { promote: false } } peers { permanent_uuid: "c88b3807d6bd413bb876555a253b8680" member_type: VOTER last_known_addr { host: "127.21.12.188" port: 35289 } attrs { promote: false } } }
I20260806 06:26:49.658967 31515 sys_catalog.cc:458] T 00000000000000000000000000000000 P c88b3807d6bd413bb876555a253b8680 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 06:26:49.659318 31400 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2589a7a56e404deda1060eb408ebf54c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2589a7a56e404deda1060eb408ebf54c" committed_config { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "2589a7a56e404deda1060eb408ebf54c" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 41635 } } peers { permanent_uuid: "124d842e9e484f8f85ddbc038dc51a31" member_type: VOTER last_known_addr { host: "127.21.12.189" port: 35823 } attrs { promote: false } } peers { permanent_uuid: "c88b3807d6bd413bb876555a253b8680" member_type: VOTER last_known_addr { host: "127.21.12.188" port: 35289 } attrs { promote: false } } }
I20260806 06:26:49.659402 31400 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2589a7a56e404deda1060eb408ebf54c [sys.catalog]: This master's current role is: LEADER
I20260806 06:26:49.659690 31181 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 124d842e9e484f8f85ddbc038dc51a31 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 7 to 10, 124d842e9e484f8f85ddbc038dc51a31 (127.21.12.189) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "2589a7a56e404deda1060eb408ebf54c" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 41635 } } peers { permanent_uuid: "124d842e9e484f8f85ddbc038dc51a31" member_type: VOTER last_known_addr { host: "127.21.12.189" port: 35823 } attrs { promote: false } } peers { permanent_uuid: "c88b3807d6bd413bb876555a253b8680" member_type: VOTER last_known_addr { host: "127.21.12.188" port: 35289 } attrs { promote: false } } }
I20260806 06:26:49.660581 32196 sys_catalog.cc:455] T 00000000000000000000000000000000 P 124d842e9e484f8f85ddbc038dc51a31 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2589a7a56e404deda1060eb408ebf54c" committed_config { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "2589a7a56e404deda1060eb408ebf54c" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 41635 } } peers { permanent_uuid: "124d842e9e484f8f85ddbc038dc51a31" member_type: VOTER last_known_addr { host: "127.21.12.189" port: 35823 } attrs { promote: false } } peers { permanent_uuid: "c88b3807d6bd413bb876555a253b8680" member_type: VOTER last_known_addr { host: "127.21.12.188" port: 35289 } attrs { promote: false } } }
I20260806 06:26:49.660663 32196 sys_catalog.cc:458] T 00000000000000000000000000000000 P 124d842e9e484f8f85ddbc038dc51a31 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 06:26:49.663517 32204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:49.658708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.130:45263 (local address 127.0.0.1:51970)
0806 06:26:49.659236 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0806 06:26:49.659245 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0806 06:26:49.659290 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:26:49.659647 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:26:49.659648 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:26:49.659651 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 06:26:49.659736 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:49.659738 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:49.661183 (+ 1445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:49.661184 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:49.662360 (+ 1176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:49.662363 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:49.663272 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:49.663274 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:49.663336 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:49.663341 (+ 5us) client_negotiation.cc:769] Sending connection context
0806 06:26:49.663398 (+ 57us) client_negotiation.cc:240] Negotiation successful
0806 06:26:49.663443 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":506,"mutex_wait_us":20,"thread_start_us":118,"threads_started":1}
I20260806 06:26:49.663546 32201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:49.658461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51970 (local address 127.21.12.130:45263)
0806 06:26:49.659418 (+ 957us) server_negotiation.cc:206] Beginning negotiation
0806 06:26:49.659423 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 06:26:49.659434 (+ 11us) server_negotiation.cc:407] Connection header received
0806 06:26:49.659475 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:26:49.659477 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:26:49.659494 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:26:49.659570 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 06:26:49.659813 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:49.661087 (+ 1274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:49.662443 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:49.663179 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:49.663255 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:49.663332 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0806 06:26:49.663426 (+ 94us) server_negotiation.cc:299] Negotiation successful
0806 06:26:49.663466 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":916,"thread_start_us":156,"threads_started":1}
I20260806 06:26:49.663719 32184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:49.658315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.129:41895 (local address 127.0.0.1:52112)
0806 06:26:49.658445 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0806 06:26:49.658456 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0806 06:26:49.658495 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:26:49.659957 (+ 1462us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:26:49.659958 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:26:49.659959 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 06:26:49.659989 (+ 30us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:49.659990 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:49.661293 (+ 1303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:49.661294 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:49.662130 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:49.662132 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:49.663502 (+ 1370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:49.663503 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:49.663576 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:49.663579 (+ 3us) client_negotiation.cc:769] Sending connection context
0806 06:26:49.663610 (+ 31us) client_negotiation.cc:240] Negotiation successful
0806 06:26:49.663657 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112}
I20260806 06:26:49.663965 32206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:49.658851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.131:46131 (local address 127.0.0.1:46948)
0806 06:26:49.659629 (+ 778us) negotiation.cc:107] Waiting for socket to connect
0806 06:26:49.659639 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0806 06:26:49.659688 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:26:49.660060 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:26:49.660061 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:26:49.660064 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 06:26:49.660114 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:49.660115 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:49.661672 (+ 1557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:49.661673 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:49.662692 (+ 1019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:49.662694 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:49.663534 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:49.663535 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:49.663809 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:49.663815 (+ 6us) client_negotiation.cc:769] Sending connection context
0806 06:26:49.663865 (+ 50us) client_negotiation.cc:240] Negotiation successful
0806 06:26:49.663902 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":761,"thread_start_us":82,"threads_started":1}
I20260806 06:26:49.664144 32202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:49.658602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52112 (local address 127.21.12.129:41895)
0806 06:26:49.659798 (+ 1196us) server_negotiation.cc:206] Beginning negotiation
0806 06:26:49.659802 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 06:26:49.659812 (+ 10us) server_negotiation.cc:407] Connection header received
0806 06:26:49.659849 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:26:49.659851 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:26:49.659867 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:26:49.660054 (+ 187us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 06:26:49.660114 (+ 60us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:49.661224 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:49.662204 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:49.663416 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:49.663889 (+ 473us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:49.664002 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0806 06:26:49.664030 (+ 28us) server_negotiation.cc:299] Negotiation successful
0806 06:26:49.664069 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1158,"thread_start_us":94,"threads_started":1}
I20260806 06:26:49.664299 32208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:49.659126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46948 (local address 127.21.12.131:46131)
0806 06:26:49.659371 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0806 06:26:49.659376 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 06:26:49.659733 (+ 357us) server_negotiation.cc:407] Connection header received
0806 06:26:49.659889 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:26:49.659892 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:26:49.659912 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:26:49.659989 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 06:26:49.660182 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:49.661581 (+ 1399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:49.662763 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:49.663366 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:49.663421 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:49.663490 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0806 06:26:49.664122 (+ 632us) server_negotiation.cc:299] Negotiation successful
0806 06:26:49.664171 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":182,"thread_start_us":97,"threads_started":1}
I20260806 06:26:49.664428 31862 tablet_service.cc:1467] Processing CreateTablet for tablet 42f9e2bb428244b59beb6be2aa36bcbe (DEFAULT_TABLE table=test-table [id=7f201577d3e84af9a5b7259958c0546f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 06:26:49.664847 31862 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42f9e2bb428244b59beb6be2aa36bcbe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 06:26:49.664855 32055 tablet_service.cc:1467] Processing CreateTablet for tablet 42f9e2bb428244b59beb6be2aa36bcbe (DEFAULT_TABLE table=test-table [id=7f201577d3e84af9a5b7259958c0546f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 06:26:49.664858 31637 tablet_service.cc:1467] Processing CreateTablet for tablet 42f9e2bb428244b59beb6be2aa36bcbe (DEFAULT_TABLE table=test-table [id=7f201577d3e84af9a5b7259958c0546f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 06:26:49.665288 32055 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42f9e2bb428244b59beb6be2aa36bcbe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 06:26:49.665288 31637 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42f9e2bb428244b59beb6be2aa36bcbe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 06:26:49.668025 32209 tablet_bootstrap.cc:492] T 42f9e2bb428244b59beb6be2aa36bcbe P 08f0835258af4cfc8d2b541884f9ceba: Bootstrap starting.
I20260806 06:26:49.668979 32209 tablet_bootstrap.cc:654] T 42f9e2bb428244b59beb6be2aa36bcbe P 08f0835258af4cfc8d2b541884f9ceba: Neither blocks nor log segments found. Creating new log.
I20260806 06:26:49.669569 32209 log.cc:824] T 42f9e2bb428244b59beb6be2aa36bcbe P 08f0835258af4cfc8d2b541884f9ceba: Log is configured to *not* fsync() on all Append() calls
I20260806 06:26:49.670083 32211 tablet_bootstrap.cc:492] T 42f9e2bb428244b59beb6be2aa36bcbe P 79d4ece6336b48e59c3e65c1d11c3ded: Bootstrap starting.
I20260806 06:26:49.670316 32212 tablet_bootstrap.cc:492] T 42f9e2bb428244b59beb6be2aa36bcbe P 9c272304769843f1af8f68c3d16ec185: Bootstrap starting.
I20260806 06:26:49.670686 32209 tablet_bootstrap.cc:492] T 42f9e2bb428244b59beb6be2aa36bcbe P 08f0835258af4cfc8d2b541884f9ceba: No bootstrap required, opened a new log
I20260806 06:26:49.670814 32209 ts_tablet_manager.cc:1397] T 42f9e2bb428244b59beb6be2aa36bcbe P 08f0835258af4cfc8d2b541884f9ceba: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260806 06:26:49.671116 32212 tablet_bootstrap.cc:654] T 42f9e2bb428244b59beb6be2aa36bcbe P 9c272304769843f1af8f68c3d16ec185: Neither blocks nor log segments found. Creating new log.
I20260806 06:26:49.671263 32211 tablet_bootstrap.cc:654] T 42f9e2bb428244b59beb6be2aa36bcbe P 79d4ece6336b48e59c3e65c1d11c3ded: Neither blocks nor log segments found. Creating new log.
I20260806 06:26:49.671638 32212 log.cc:824] T 42f9e2bb428244b59beb6be2aa36bcbe P 9c272304769843f1af8f68c3d16ec185: Log is configured to *not* fsync() on all Append() calls
I20260806 06:26:49.671979 32211 log.cc:824] T 42f9e2bb428244b59beb6be2aa36bcbe P 79d4ece6336b48e59c3e65c1d11c3ded: Log is configured to *not* fsync() on all Append() calls
I20260806 06:26:49.672403 32212 tablet_bootstrap.cc:492] T 42f9e2bb428244b59beb6be2aa36bcbe P 9c272304769843f1af8f68c3d16ec185: No bootstrap required, opened a new log
I20260806 06:26:49.672484 32212 ts_tablet_manager.cc:1397] T 42f9e2bb428244b59beb6be2aa36bcbe P 9c272304769843f1af8f68c3d16ec185: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260806 06:26:49.672879 32211 tablet_bootstrap.cc:492] T 42f9e2bb428244b59beb6be2aa36bcbe P 79d4ece6336b48e59c3e65c1d11c3ded: No bootstrap required, opened a new log
I20260806 06:26:49.672950 32211 ts_tablet_manager.cc:1397] T 42f9e2bb428244b59beb6be2aa36bcbe P 79d4ece6336b48e59c3e65c1d11c3ded: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260806 06:26:49.674125 32209 raft_consensus.cc:348] T 42f9e2bb428244b59beb6be2aa36bcbe P 08f0835258af4cfc8d2b541884f9ceba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c272304769843f1af8f68c3d16ec185" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 41895 } } peers { permanent_uuid: "08f0835258af4cfc8d2b541884f9ceba" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 45263 } } peers { permanent_uuid: "79d4ece6336b48e59c3e65c1d11c3ded" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 46131 } }
I20260806 06:26:49.674256 32209 raft_consensus.cc:374] T 42f9e2bb428244b59beb6be2aa36bcbe P 08f0835258af4cfc8d2b541884f9ceba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 06:26:49.674306 32209 raft_consensus.cc:729] T 42f9e2bb428244b59beb6be2aa36bcbe P 08f0835258af4cfc8d2b541884f9ceba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08f0835258af4cfc8d2b541884f9ceba, State: Initialized, Role: FOLLOWER
I20260806 06:26:49.674440 32209 consensus_queue.cc:260] T 42f9e2bb428244b59beb6be2aa36bcbe P 08f0835258af4cfc8d2b541884f9ceba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c272304769843f1af8f68c3d16ec185" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 41895 } } peers { permanent_uuid: "08f0835258af4cfc8d2b541884f9ceba" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 45263 } } peers { permanent_uuid: "79d4ece6336b48e59c3e65c1d11c3ded" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 46131 } }
I20260806 06:26:49.674731 32209 ts_tablet_manager.cc:1428] T 42f9e2bb428244b59beb6be2aa36bcbe P 08f0835258af4cfc8d2b541884f9ceba: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260806 06:26:49.674813 31950 heartbeater.cc:503] Master 127.21.12.190:41635 was elected leader, sending a full tablet report...
I20260806 06:26:49.675302 32212 raft_consensus.cc:348] T 42f9e2bb428244b59beb6be2aa36bcbe P 9c272304769843f1af8f68c3d16ec185 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c272304769843f1af8f68c3d16ec185" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 41895 } } peers { permanent_uuid: "08f0835258af4cfc8d2b541884f9ceba" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 45263 } } peers { permanent_uuid: "79d4ece6336b48e59c3e65c1d11c3ded" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 46131 } }
I20260806 06:26:49.675467 32212 raft_consensus.cc:374] T 42f9e2bb428244b59beb6be2aa36bcbe P 9c272304769843f1af8f68c3d16ec185 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 06:26:49.675498 32212 raft_consensus.cc:729] T 42f9e2bb428244b59beb6be2aa36bcbe P 9c272304769843f1af8f68c3d16ec185 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c272304769843f1af8f68c3d16ec185, State: Initialized, Role: FOLLOWER
I20260806 06:26:49.675598 32212 consensus_queue.cc:260] T 42f9e2bb428244b59beb6be2aa36bcbe P 9c272304769843f1af8f68c3d16ec185 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c272304769843f1af8f68c3d16ec185" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 41895 } } peers { permanent_uuid: "08f0835258af4cfc8d2b541884f9ceba" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 45263 } } peers { permanent_uuid: "79d4ece6336b48e59c3e65c1d11c3ded" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 46131 } }
I20260806 06:26:49.675906 32212 ts_tablet_manager.cc:1428] T 42f9e2bb428244b59beb6be2aa36bcbe P 9c272304769843f1af8f68c3d16ec185: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260806 06:26:49.676018 31756 heartbeater.cc:503] Master 127.21.12.190:41635 was elected leader, sending a full tablet report...
I20260806 06:26:49.676712 32211 raft_consensus.cc:348] T 42f9e2bb428244b59beb6be2aa36bcbe P 79d4ece6336b48e59c3e65c1d11c3ded [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c272304769843f1af8f68c3d16ec185" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 41895 } } peers { permanent_uuid: "08f0835258af4cfc8d2b541884f9ceba" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 45263 } } peers { permanent_uuid: "79d4ece6336b48e59c3e65c1d11c3ded" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 46131 } }
I20260806 06:26:49.676864 32211 raft_consensus.cc:374] T 42f9e2bb428244b59beb6be2aa36bcbe P 79d4ece6336b48e59c3e65c1d11c3ded [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 06:26:49.676903 32211 raft_consensus.cc:729] T 42f9e2bb428244b59beb6be2aa36bcbe P 79d4ece6336b48e59c3e65c1d11c3ded [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79d4ece6336b48e59c3e65c1d11c3ded, State: Initialized, Role: FOLLOWER
I20260806 06:26:49.677030 32211 consensus_queue.cc:260] T 42f9e2bb428244b59beb6be2aa36bcbe P 79d4ece6336b48e59c3e65c1d11c3ded [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c272304769843f1af8f68c3d16ec185" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 41895 } } peers { permanent_uuid: "08f0835258af4cfc8d2b541884f9ceba" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 45263 } } peers { permanent_uuid: "79d4ece6336b48e59c3e65c1d11c3ded" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 46131 } }
I20260806 06:26:49.677345 32211 ts_tablet_manager.cc:1428] T 42f9e2bb428244b59beb6be2aa36bcbe P 79d4ece6336b48e59c3e65c1d11c3ded: Time spent starting tablet: real 0.004s user 0.001s sys 0.003s
I20260806 06:26:49.677431 32154 heartbeater.cc:503] Master 127.21.12.190:41635 was elected leader, sending a full tablet report...
W20260806 06:26:49.680743 31952 tablet.cc:2369] T 42f9e2bb428244b59beb6be2aa36bcbe P 08f0835258af4cfc8d2b541884f9ceba: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 06:26:49.763909 32219 raft_consensus.cc:482] T 42f9e2bb428244b59beb6be2aa36bcbe P 79d4ece6336b48e59c3e65c1d11c3ded [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 06:26:49.764043 32219 raft_consensus.cc:504] T 42f9e2bb428244b59beb6be2aa36bcbe P 79d4ece6336b48e59c3e65c1d11c3ded [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c272304769843f1af8f68c3d16ec185" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 41895 } } peers { permanent_uuid: "08f0835258af4cfc8d2b541884f9ceba" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 45263 } } peers { permanent_uuid: "79d4ece6336b48e59c3e65c1d11c3ded" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 46131 } }
I20260806 06:26:49.764461 32219 leader_election.cc:288] T 42f9e2bb428244b59beb6be2aa36bcbe P 79d4ece6336b48e59c3e65c1d11c3ded [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9c272304769843f1af8f68c3d16ec185 (127.21.12.129:41895), 08f0835258af4cfc8d2b541884f9ceba (127.21.12.130:45263)
I20260806 06:26:49.767289 32202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:49.764769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.131:46373 (local address 127.21.12.129:41895)
0806 06:26:49.764811 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0806 06:26:49.764812 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0806 06:26:49.764820 (+ 8us) server_negotiation.cc:407] Connection header received
0806 06:26:49.764843 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:26:49.764844 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:26:49.764850 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:26:49.764902 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 06:26:49.765052 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:49.765550 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:49.766591 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:49.766938 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:49.766993 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:49.767000 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0806 06:26:49.767202 (+ 202us) server_negotiation.cc:299] Negotiation successful
0806 06:26:49.767236 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260806 06:26:49.767315 32161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:49.764644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.129:41895 (local address 127.21.12.131:46373)
0806 06:26:49.764696 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0806 06:26:49.764706 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0806 06:26:49.764747 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:26:49.764912 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:26:49.764913 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:26:49.764914 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 06:26:49.764974 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:49.764975 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:49.765623 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:49.765624 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:49.766508 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:49.766511 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:49.767014 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:49.767015 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:49.767133 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:49.767137 (+ 4us) client_negotiation.cc:769] Sending connection context
0806 06:26:49.767191 (+ 54us) client_negotiation.cc:240] Negotiation successful
0806 06:26:49.767235 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36,"mutex_wait_us":39}
I20260806 06:26:49.767529 32157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:49.764634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.130:45263 (local address 127.21.12.131:48731)
0806 06:26:49.764682 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0806 06:26:49.764695 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 06:26:49.764741 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:26:49.764890 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:26:49.764890 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:26:49.764892 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 06:26:49.765040 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:49.765042 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:49.765676 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:49.765676 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:49.766483 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:49.766485 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:49.767255 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:49.767255 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:49.767385 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:49.767388 (+ 3us) client_negotiation.cc:769] Sending connection context
0806 06:26:49.767437 (+ 49us) client_negotiation.cc:240] Negotiation successful
0806 06:26:49.767475 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28,"mutex_wait_us":2}
I20260806 06:26:49.767738 32201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:49.764735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.131:48731 (local address 127.21.12.130:45263)
0806 06:26:49.764786 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0806 06:26:49.764788 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:26:49.764796 (+ 8us) server_negotiation.cc:407] Connection header received
0806 06:26:49.764822 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:26:49.764823 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:26:49.764830 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:26:49.764874 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 06:26:49.765123 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:49.765600 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:49.766572 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:49.767128 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:49.767202 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:49.767211 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0806 06:26:49.767631 (+ 420us) server_negotiation.cc:299] Negotiation successful
0806 06:26:49.767680 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260806 06:26:49.767728 31679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42f9e2bb428244b59beb6be2aa36bcbe" candidate_uuid: "79d4ece6336b48e59c3e65c1d11c3ded" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c272304769843f1af8f68c3d16ec185" is_pre_election: true
I20260806 06:26:49.767882 31679 raft_consensus.cc:2454] T 42f9e2bb428244b59beb6be2aa36bcbe P 9c272304769843f1af8f68c3d16ec185 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 79d4ece6336b48e59c3e65c1d11c3ded in term 0.
I20260806 06:26:49.768180 31887 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42f9e2bb428244b59beb6be2aa36bcbe" candidate_uuid: "79d4ece6336b48e59c3e65c1d11c3ded" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08f0835258af4cfc8d2b541884f9ceba" is_pre_election: true
I20260806 06:26:49.768314 31887 raft_consensus.cc:2454] T 42f9e2bb428244b59beb6be2aa36bcbe P 08f0835258af4cfc8d2b541884f9ceba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 79d4ece6336b48e59c3e65c1d11c3ded in term 0.
I20260806 06:26:49.768471 31984 leader_election.cc:302] T 42f9e2bb428244b59beb6be2aa36bcbe P 79d4ece6336b48e59c3e65c1d11c3ded [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 79d4ece6336b48e59c3e65c1d11c3ded, 9c272304769843f1af8f68c3d16ec185; no voters:
I20260806 06:26:49.768613 32219 raft_consensus.cc:2781] T 42f9e2bb428244b59beb6be2aa36bcbe P 79d4ece6336b48e59c3e65c1d11c3ded [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 06:26:49.768671 32219 raft_consensus.cc:482] T 42f9e2bb428244b59beb6be2aa36bcbe P 79d4ece6336b48e59c3e65c1d11c3ded [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 06:26:49.768694 32219 raft_consensus.cc:3037] T 42f9e2bb428244b59beb6be2aa36bcbe P 79d4ece6336b48e59c3e65c1d11c3ded [term 0 FOLLOWER]: Advancing to term 1
I20260806 06:26:49.769663 32219 raft_consensus.cc:504] T 42f9e2bb428244b59beb6be2aa36bcbe P 79d4ece6336b48e59c3e65c1d11c3ded [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c272304769843f1af8f68c3d16ec185" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 41895 } } peers { permanent_uuid: "08f0835258af4cfc8d2b541884f9ceba" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 45263 } } peers { permanent_uuid: "79d4ece6336b48e59c3e65c1d11c3ded" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 46131 } }
I20260806 06:26:49.769776 32219 leader_election.cc:288] T 42f9e2bb428244b59beb6be2aa36bcbe P 79d4ece6336b48e59c3e65c1d11c3ded [CANDIDATE]: Term 1 election: Requested vote from peers 9c272304769843f1af8f68c3d16ec185 (127.21.12.129:41895), 08f0835258af4cfc8d2b541884f9ceba (127.21.12.130:45263)
I20260806 06:26:49.769924 31679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42f9e2bb428244b59beb6be2aa36bcbe" candidate_uuid: "79d4ece6336b48e59c3e65c1d11c3ded" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c272304769843f1af8f68c3d16ec185"
I20260806 06:26:49.769924 31887 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42f9e2bb428244b59beb6be2aa36bcbe" candidate_uuid: "79d4ece6336b48e59c3e65c1d11c3ded" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08f0835258af4cfc8d2b541884f9ceba"
I20260806 06:26:49.770012 31679 raft_consensus.cc:3037] T 42f9e2bb428244b59beb6be2aa36bcbe P 9c272304769843f1af8f68c3d16ec185 [term 0 FOLLOWER]: Advancing to term 1
I20260806 06:26:49.770097 31887 raft_consensus.cc:3037] T 42f9e2bb428244b59beb6be2aa36bcbe P 08f0835258af4cfc8d2b541884f9ceba [term 0 FOLLOWER]: Advancing to term 1
I20260806 06:26:49.771205 31887 raft_consensus.cc:2454] T 42f9e2bb428244b59beb6be2aa36bcbe P 08f0835258af4cfc8d2b541884f9ceba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 79d4ece6336b48e59c3e65c1d11c3ded in term 1.
I20260806 06:26:49.771406 31981 leader_election.cc:302] T 42f9e2bb428244b59beb6be2aa36bcbe P 79d4ece6336b48e59c3e65c1d11c3ded [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08f0835258af4cfc8d2b541884f9ceba, 79d4ece6336b48e59c3e65c1d11c3ded; no voters:
I20260806 06:26:49.771513 32219 raft_consensus.cc:2781] T 42f9e2bb428244b59beb6be2aa36bcbe P 79d4ece6336b48e59c3e65c1d11c3ded [term 1 FOLLOWER]: Leader election won for term 1
I20260806 06:26:49.771625 31679 raft_consensus.cc:2454] T 42f9e2bb428244b59beb6be2aa36bcbe P 9c272304769843f1af8f68c3d16ec185 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 79d4ece6336b48e59c3e65c1d11c3ded in term 1.
I20260806 06:26:49.771719 32219 raft_consensus.cc:686] T 42f9e2bb428244b59beb6be2aa36bcbe P 79d4ece6336b48e59c3e65c1d11c3ded [term 1 LEADER]: Becoming Leader. State: Replica: 79d4ece6336b48e59c3e65c1d11c3ded, State: Running, Role: LEADER
I20260806 06:26:49.771873 32219 consensus_queue.cc:237] T 42f9e2bb428244b59beb6be2aa36bcbe P 79d4ece6336b48e59c3e65c1d11c3ded [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c272304769843f1af8f68c3d16ec185" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 41895 } } peers { permanent_uuid: "08f0835258af4cfc8d2b541884f9ceba" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 45263 } } peers { permanent_uuid: "79d4ece6336b48e59c3e65c1d11c3ded" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 46131 } }
I20260806 06:26:49.772698 30217 catalog_manager.cc:5495] T 42f9e2bb428244b59beb6be2aa36bcbe P 79d4ece6336b48e59c3e65c1d11c3ded reported cstate change: term changed from 0 to 1, leader changed from <none> to 79d4ece6336b48e59c3e65c1d11c3ded (127.21.12.131). New cstate: current_term: 1 leader_uuid: "79d4ece6336b48e59c3e65c1d11c3ded" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c272304769843f1af8f68c3d16ec185" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 41895 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "08f0835258af4cfc8d2b541884f9ceba" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 45263 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "79d4ece6336b48e59c3e65c1d11c3ded" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 46131 } health_report { overall_health: HEALTHY } } }
I20260806 06:26:49.780028 32208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:49.777829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46964 (local address 127.21.12.131:46131)
0806 06:26:49.777869 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0806 06:26:49.777871 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:26:49.777878 (+ 7us) server_negotiation.cc:407] Connection header received
0806 06:26:49.777943 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:26:49.777943 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:26:49.777961 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:26:49.778013 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 06:26:49.778120 (+ 107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:49.778625 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:49.779349 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:49.779547 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:49.779601 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:49.779743 (+ 142us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 06:26:49.779835 (+ 92us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 06:26:49.779902 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0806 06:26:49.779939 (+ 37us) server_negotiation.cc:299] Negotiation successful
0806 06:26:49.779958 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
W20260806 06:26:49.780829 31759 tablet.cc:2369] T 42f9e2bb428244b59beb6be2aa36bcbe P 9c272304769843f1af8f68c3d16ec185: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 06:26:49.782078 31887 raft_consensus.cc:1261] T 42f9e2bb428244b59beb6be2aa36bcbe P 08f0835258af4cfc8d2b541884f9ceba [term 1 FOLLOWER]: Refusing update from remote peer 79d4ece6336b48e59c3e65c1d11c3ded: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 06:26:49.782176 31679 raft_consensus.cc:1261] T 42f9e2bb428244b59beb6be2aa36bcbe P 9c272304769843f1af8f68c3d16ec185 [term 1 FOLLOWER]: Refusing update from remote peer 79d4ece6336b48e59c3e65c1d11c3ded: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 06:26:49.782478 32220 consensus_queue.cc:1035] T 42f9e2bb428244b59beb6be2aa36bcbe P 79d4ece6336b48e59c3e65c1d11c3ded [LEADER]: Connected to new peer: Peer: permanent_uuid: "08f0835258af4cfc8d2b541884f9ceba" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 45263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 06:26:49.782605 32219 consensus_queue.cc:1035] T 42f9e2bb428244b59beb6be2aa36bcbe P 79d4ece6336b48e59c3e65c1d11c3ded [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c272304769843f1af8f68c3d16ec185" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 41895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 06:26:49.785776 32227 mvcc.cc:205] Tried to move back new op lower bound from 7315446209665044480 to 7315446209626304512. Current Snapshot: MvccSnapshot[applied={T|T < 7315446209665044480}]
I20260806 06:26:49.788683 30216 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50786:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 06:26:49.788780 30216 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50786:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 06:26:49.790328 30216 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2589a7a56e404deda1060eb408ebf54c: Sending DeleteTablet for 3 replicas of tablet 42f9e2bb428244b59beb6be2aa36bcbe
I20260806 06:26:49.790665 31637 tablet_service.cc:1514] Processing DeleteTablet for tablet 42f9e2bb428244b59beb6be2aa36bcbe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 06:26:49 UTC) from {username='oryx', principal='oryx/127.21.12.190@KRBTEST.COM'} at 127.0.0.1:52112
I20260806 06:26:49.790840 32055 tablet_service.cc:1514] Processing DeleteTablet for tablet 42f9e2bb428244b59beb6be2aa36bcbe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 06:26:49 UTC) from {username='oryx', principal='oryx/127.21.12.190@KRBTEST.COM'} at 127.0.0.1:46948
I20260806 06:26:49.790930 31862 tablet_service.cc:1514] Processing DeleteTablet for tablet 42f9e2bb428244b59beb6be2aa36bcbe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 06:26:49 UTC) from {username='oryx', principal='oryx/127.21.12.190@KRBTEST.COM'} at 127.0.0.1:51970
I20260806 06:26:49.791225 32244 tablet_replica.cc:330] T 42f9e2bb428244b59beb6be2aa36bcbe P 08f0835258af4cfc8d2b541884f9ceba: stopping tablet replica
I20260806 06:26:49.791337 32244 raft_consensus.cc:2229] T 42f9e2bb428244b59beb6be2aa36bcbe P 08f0835258af4cfc8d2b541884f9ceba [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 06:26:49.791424 32242 tablet_replica.cc:330] T 42f9e2bb428244b59beb6be2aa36bcbe P 9c272304769843f1af8f68c3d16ec185: stopping tablet replica
I20260806 06:26:49.791443 32244 raft_consensus.cc:2258] T 42f9e2bb428244b59beb6be2aa36bcbe P 08f0835258af4cfc8d2b541884f9ceba [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 06:26:49.791538 32242 raft_consensus.cc:2229] T 42f9e2bb428244b59beb6be2aa36bcbe P 9c272304769843f1af8f68c3d16ec185 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 06:26:49.791653 32242 raft_consensus.cc:2258] T 42f9e2bb428244b59beb6be2aa36bcbe P 9c272304769843f1af8f68c3d16ec185 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 06:26:49.791923 32244 ts_tablet_manager.cc:1905] T 42f9e2bb428244b59beb6be2aa36bcbe P 08f0835258af4cfc8d2b541884f9ceba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 06:26:49.792490 21554 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHhLhT/build/release/bin/kudu with pid 30163
I20260806 06:26:49.797981 32245 tablet_replica.cc:330] T 42f9e2bb428244b59beb6be2aa36bcbe P 79d4ece6336b48e59c3e65c1d11c3ded: stopping tablet replica
I20260806 06:26:49.798069 32244 ts_tablet_manager.cc:1918] T 42f9e2bb428244b59beb6be2aa36bcbe P 08f0835258af4cfc8d2b541884f9ceba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 06:26:49.798148 32245 raft_consensus.cc:2229] T 42f9e2bb428244b59beb6be2aa36bcbe P 79d4ece6336b48e59c3e65c1d11c3ded [term 1 LEADER]: Raft consensus shutting down.
I20260806 06:26:49.798168 32244 log.cc:1196] T 42f9e2bb428244b59beb6be2aa36bcbe P 08f0835258af4cfc8d2b541884f9ceba: Deleting WAL directory at /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785997581049705-21554-0/minicluster-data/ts-1/wal/wals/42f9e2bb428244b59beb6be2aa36bcbe
I20260806 06:26:49.798327 32245 raft_consensus.cc:2258] T 42f9e2bb428244b59beb6be2aa36bcbe P 79d4ece6336b48e59c3e65c1d11c3ded [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 06:26:49.798488 32244 ts_tablet_manager.cc:1939] T 42f9e2bb428244b59beb6be2aa36bcbe P 08f0835258af4cfc8d2b541884f9ceba: Deleting consensus metadata
I20260806 06:26:49.800560 32245 ts_tablet_manager.cc:1905] T 42f9e2bb428244b59beb6be2aa36bcbe P 79d4ece6336b48e59c3e65c1d11c3ded: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 06:26:49.802510 32245 ts_tablet_manager.cc:1918] T 42f9e2bb428244b59beb6be2aa36bcbe P 79d4ece6336b48e59c3e65c1d11c3ded: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 06:26:49.802590 32245 log.cc:1196] T 42f9e2bb428244b59beb6be2aa36bcbe P 79d4ece6336b48e59c3e65c1d11c3ded: Deleting WAL directory at /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785997581049705-21554-0/minicluster-data/ts-2/wal/wals/42f9e2bb428244b59beb6be2aa36bcbe
I20260806 06:26:49.802847 32245 ts_tablet_manager.cc:1939] T 42f9e2bb428244b59beb6be2aa36bcbe P 79d4ece6336b48e59c3e65c1d11c3ded: Deleting consensus metadata
I20260806 06:26:49.805612 31758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:49.791594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35348 (local address 127.21.12.188:35289)
0806 06:26:49.791634 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0806 06:26:49.791636 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:26:49.792157 (+ 521us) server_negotiation.cc:407] Connection header received
0806 06:26:49.792355 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:26:49.792356 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:26:49.792366 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:26:49.792396 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:26:49.795583 (+ 3187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:49.796554 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:49.799682 (+ 3128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:49.800073 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:49.800125 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:49.802344 (+ 2219us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:26:49.802346 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:26:49.802346 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 06:26:49.802354 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:26:49.803997 (+ 1643us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:26:49.804515 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:26:49.804516 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:26:49.804517 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:26:49.804589 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:26:49.804865 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:26:49.804867 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:26:49.804867 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:26:49.805027 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:26:49.805121 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0806 06:26:49.805393 (+ 272us) server_negotiation.cc:299] Negotiation successful
0806 06:26:49.805524 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260806 06:26:49.809729 32242 ts_tablet_manager.cc:1905] T 42f9e2bb428244b59beb6be2aa36bcbe P 9c272304769843f1af8f68c3d16ec185: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260806 06:26:49.810650 31785 connection.cc:535] server connection from 127.0.0.1:51970 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260806 06:26:49.810882 31983 connection.cc:535] server connection from 127.0.0.1:46948 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260806 06:26:49.811643 32242 ts_tablet_manager.cc:1918] T 42f9e2bb428244b59beb6be2aa36bcbe P 9c272304769843f1af8f68c3d16ec185: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 06:26:49.811730 32242 log.cc:1196] T 42f9e2bb428244b59beb6be2aa36bcbe P 9c272304769843f1af8f68c3d16ec185: Deleting WAL directory at /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785997581049705-21554-0/minicluster-data/ts-0/wal/wals/42f9e2bb428244b59beb6be2aa36bcbe
I20260806 06:26:49.811998 32242 ts_tablet_manager.cc:1939] T 42f9e2bb428244b59beb6be2aa36bcbe P 9c272304769843f1af8f68c3d16ec185: Deleting consensus metadata
W20260806 06:26:49.812366 31560 connection.cc:441] server connection from 127.0.0.1:52112 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52112 (request call id 1) could send its response
I20260806 06:26:49.820420 31758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:49.813063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35354 (local address 127.21.12.188:35289)
0806 06:26:49.813105 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0806 06:26:49.813107 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:26:49.813231 (+ 124us) server_negotiation.cc:407] Connection header received
0806 06:26:49.813306 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:26:49.813307 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:26:49.813317 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:26:49.813381 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:26:49.813948 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:49.814790 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:49.815695 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:49.815995 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:49.816070 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:49.817651 (+ 1581us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:26:49.817653 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:26:49.817653 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 06:26:49.817662 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:26:49.819256 (+ 1594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:26:49.819646 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:26:49.819647 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:26:49.819648 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:26:49.819711 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:26:49.819906 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:26:49.819907 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:26:49.819907 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:26:49.820038 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:26:49.820125 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0806 06:26:49.820253 (+ 128us) server_negotiation.cc:299] Negotiation successful
0806 06:26:49.820331 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260806 06:26:49.820472 32172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:49.812761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59490 (local address 127.21.12.189:35823)
0806 06:26:49.812808 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0806 06:26:49.812810 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:26:49.813016 (+ 206us) server_negotiation.cc:407] Connection header received
0806 06:26:49.813121 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:26:49.813122 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:26:49.813137 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:26:49.813170 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:26:49.813921 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:49.814503 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:49.815380 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:49.815696 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:49.815732 (+ 36us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:49.816945 (+ 1213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:26:49.816947 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:26:49.816947 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 06:26:49.816955 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:26:49.818510 (+ 1555us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:26:49.819679 (+ 1169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:26:49.819680 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:26:49.819680 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:26:49.819732 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:26:49.819919 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:26:49.819920 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:26:49.819920 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:26:49.820039 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:26:49.820112 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0806 06:26:49.820317 (+ 205us) server_negotiation.cc:299] Negotiation successful
0806 06:26:49.820393 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260806 06:26:50.787039 32257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:50.786697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:41635 (local address 127.21.12.129:48543)
0806 06:26:50.786896 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0806 06:26:50.786983 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.190:41635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":87,"threads_started":1}
W20260806 06:26:50.787300 31756 heartbeater.cc:644] Failed to heartbeat to 127.21.12.190:41635 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.21.12.190:41635: connect: Connection refused (error 111)
I20260806 06:26:50.787544 32259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:50.786914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:41635 (local address 127.21.12.130:56741)
0806 06:26:50.787019 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0806 06:26:50.787090 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.190:41635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88,"thread_start_us":51,"threads_started":1}
I20260806 06:26:50.787621 32260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:50.787164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:41635 (local address 127.21.12.131:47613)
0806 06:26:50.787512 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0806 06:26:50.787580 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.190:41635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":48,"threads_started":1}
W20260806 06:26:50.787813 32154 heartbeater.cc:644] Failed to heartbeat to 127.21.12.190:41635 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.21.12.190:41635: connect: Connection refused (error 111)
W20260806 06:26:50.787884 31950 heartbeater.cc:644] Failed to heartbeat to 127.21.12.190:41635 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.21.12.190:41635: connect: Connection refused (error 111)
I20260806 06:26:51.290755 32263 raft_consensus.cc:482] T 00000000000000000000000000000000 P c88b3807d6bd413bb876555a253b8680 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2589a7a56e404deda1060eb408ebf54c)
I20260806 06:26:51.290884 32263 raft_consensus.cc:504] T 00000000000000000000000000000000 P c88b3807d6bd413bb876555a253b8680 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "2589a7a56e404deda1060eb408ebf54c" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 41635 } } peers { permanent_uuid: "124d842e9e484f8f85ddbc038dc51a31" member_type: VOTER last_known_addr { host: "127.21.12.189" port: 35823 } attrs { promote: false } } peers { permanent_uuid: "c88b3807d6bd413bb876555a253b8680" member_type: VOTER last_known_addr { host: "127.21.12.188" port: 35289 } attrs { promote: false } }
I20260806 06:26:51.291158 32263 leader_election.cc:288] T 00000000000000000000000000000000 P c88b3807d6bd413bb876555a253b8680 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2589a7a56e404deda1060eb408ebf54c (127.21.12.190:41635), 124d842e9e484f8f85ddbc038dc51a31 (127.21.12.189:35823)
I20260806 06:26:51.291139 32264 raft_consensus.cc:482] T 00000000000000000000000000000000 P 124d842e9e484f8f85ddbc038dc51a31 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2589a7a56e404deda1060eb408ebf54c)
I20260806 06:26:51.291275 32264 raft_consensus.cc:504] T 00000000000000000000000000000000 P 124d842e9e484f8f85ddbc038dc51a31 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "2589a7a56e404deda1060eb408ebf54c" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 41635 } } peers { permanent_uuid: "124d842e9e484f8f85ddbc038dc51a31" member_type: VOTER last_known_addr { host: "127.21.12.189" port: 35823 } attrs { promote: false } } peers { permanent_uuid: "c88b3807d6bd413bb876555a253b8680" member_type: VOTER last_known_addr { host: "127.21.12.188" port: 35289 } attrs { promote: false } }
I20260806 06:26:51.291489 32265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:51.291289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:41635 (local address 127.0.0.1:51234)
0806 06:26:51.291395 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0806 06:26:51.291447 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.190:41635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":46,"threads_started":1}
I20260806 06:26:51.291525 32264 leader_election.cc:288] T 00000000000000000000000000000000 P 124d842e9e484f8f85ddbc038dc51a31 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2589a7a56e404deda1060eb408ebf54c (127.21.12.190:41635), c88b3807d6bd413bb876555a253b8680 (127.21.12.188:35289)
W20260806 06:26:51.291599 31434 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.12.190:41635: connect: Connection refused (error 111)
I20260806 06:26:51.291955 32272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:51.291621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:41635 (local address 127.0.0.1:51250)
0806 06:26:51.291852 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0806 06:26:51.291917 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.190:41635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":49,"threads_started":1}
I20260806 06:26:51.292017 32265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:51.291900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:41635 (local address 127.0.0.1:51254)
0806 06:26:51.291963 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0806 06:26:51.291989 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.190:41635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53,"thread_start_us":65,"threads_started":1}
W20260806 06:26:51.292110 31434 leader_election.cc:334] T 00000000000000000000000000000000 P c88b3807d6bd413bb876555a253b8680 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2589a7a56e404deda1060eb408ebf54c (127.21.12.190:41635): Network error: Client connection negotiation failed: client connection to 127.21.12.190:41635: connect: Connection refused (error 111)
W20260806 06:26:51.292246 31150 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.12.190:41635: connect: Connection refused (error 111)
I20260806 06:26:51.292629 32272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:51.292541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:41635 (local address 127.0.0.1:51256)
0806 06:26:51.292587 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0806 06:26:51.292610 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.190:41635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260806 06:26:51.292725 31150 leader_election.cc:334] T 00000000000000000000000000000000 P 124d842e9e484f8f85ddbc038dc51a31 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2589a7a56e404deda1060eb408ebf54c (127.21.12.190:41635): Network error: Client connection negotiation failed: client connection to 127.21.12.190:41635: connect: Connection refused (error 111)
I20260806 06:26:51.295205 32269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:51.291612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59506 (local address 127.21.12.189:35823)
0806 06:26:51.291711 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0806 06:26:51.291713 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:26:51.291723 (+ 10us) server_negotiation.cc:407] Connection header received
0806 06:26:51.291917 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:26:51.291918 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:26:51.291931 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:26:51.291984 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 06:26:51.292194 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:51.292983 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:51.294380 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:51.294918 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:51.294989 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:51.295001 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0806 06:26:51.295115 (+ 114us) server_negotiation.cc:299] Negotiation successful
0806 06:26:51.295146 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81,"thread_start_us":46,"threads_started":1}
I20260806 06:26:51.295219 32266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:51.291327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.189:35823 (local address 127.0.0.1:59506)
0806 06:26:51.291424 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0806 06:26:51.291434 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0806 06:26:51.291497 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:26:51.292039 (+ 542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:26:51.292040 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:26:51.292043 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 06:26:51.292113 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:51.292115 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:51.293058 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:51.293059 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:51.294258 (+ 1199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:51.294261 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:51.294990 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:51.294991 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:51.295043 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:51.295049 (+ 6us) client_negotiation.cc:769] Sending connection context
0806 06:26:51.295108 (+ 59us) client_negotiation.cc:240] Negotiation successful
0806 06:26:51.295153 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":85,"thread_start_us":65,"threads_started":1}
I20260806 06:26:51.295379 32273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:51.291858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35368 (local address 127.21.12.188:35289)
0806 06:26:51.292052 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0806 06:26:51.292054 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:26:51.292242 (+ 188us) server_negotiation.cc:407] Connection header received
0806 06:26:51.292262 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:26:51.292263 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:26:51.292272 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:26:51.292324 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 06:26:51.292452 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:51.293155 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:51.294475 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:51.294994 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:51.295051 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:51.295067 (+ 16us) server_negotiation.cc:1035] Waiting for connection context
0806 06:26:51.295174 (+ 107us) server_negotiation.cc:299] Negotiation successful
0806 06:26:51.295211 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":73,"threads_started":1}
I20260806 06:26:51.295463 32270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:51.291769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.188:35289 (local address 127.0.0.1:35368)
0806 06:26:51.292139 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0806 06:26:51.292149 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0806 06:26:51.292198 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:26:51.292357 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:26:51.292357 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:26:51.292359 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 06:26:51.292391 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:51.292392 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:51.293223 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:51.293225 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:51.294412 (+ 1187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:51.294415 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:51.295058 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:51.295059 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:51.295116 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:51.295119 (+ 3us) client_negotiation.cc:769] Sending connection context
0806 06:26:51.295165 (+ 46us) client_negotiation.cc:240] Negotiation successful
0806 06:26:51.295207 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":52,"threads_started":1}
I20260806 06:26:51.295426 31181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c88b3807d6bd413bb876555a253b8680" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "124d842e9e484f8f85ddbc038dc51a31" is_pre_election: true
I20260806 06:26:51.295542 31181 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 124d842e9e484f8f85ddbc038dc51a31 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c88b3807d6bd413bb876555a253b8680 in term 1.
I20260806 06:26:51.295697 31433 leader_election.cc:302] T 00000000000000000000000000000000 P c88b3807d6bd413bb876555a253b8680 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 124d842e9e484f8f85ddbc038dc51a31, c88b3807d6bd413bb876555a253b8680; no voters: 2589a7a56e404deda1060eb408ebf54c
I20260806 06:26:51.295712 31466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "124d842e9e484f8f85ddbc038dc51a31" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "c88b3807d6bd413bb876555a253b8680" is_pre_election: true
I20260806 06:26:51.295821 31466 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c88b3807d6bd413bb876555a253b8680 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 124d842e9e484f8f85ddbc038dc51a31 in term 1.
I20260806 06:26:51.295897 32263 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c88b3807d6bd413bb876555a253b8680 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 06:26:51.295921 32263 raft_consensus.cc:482] T 00000000000000000000000000000000 P c88b3807d6bd413bb876555a253b8680 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 2589a7a56e404deda1060eb408ebf54c)
I20260806 06:26:51.295939 32263 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c88b3807d6bd413bb876555a253b8680 [term 1 FOLLOWER]: Advancing to term 2
I20260806 06:26:51.295948 31150 leader_election.cc:302] T 00000000000000000000000000000000 P 124d842e9e484f8f85ddbc038dc51a31 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 124d842e9e484f8f85ddbc038dc51a31, c88b3807d6bd413bb876555a253b8680; no voters: 2589a7a56e404deda1060eb408ebf54c
I20260806 06:26:51.296036 32264 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 124d842e9e484f8f85ddbc038dc51a31 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 06:26:51.296084 32264 raft_consensus.cc:482] T 00000000000000000000000000000000 P 124d842e9e484f8f85ddbc038dc51a31 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 2589a7a56e404deda1060eb408ebf54c)
I20260806 06:26:51.296111 32264 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 124d842e9e484f8f85ddbc038dc51a31 [term 1 FOLLOWER]: Advancing to term 2
I20260806 06:26:51.296644 32263 raft_consensus.cc:504] T 00000000000000000000000000000000 P c88b3807d6bd413bb876555a253b8680 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "2589a7a56e404deda1060eb408ebf54c" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 41635 } } peers { permanent_uuid: "124d842e9e484f8f85ddbc038dc51a31" member_type: VOTER last_known_addr { host: "127.21.12.189" port: 35823 } attrs { promote: false } } peers { permanent_uuid: "c88b3807d6bd413bb876555a253b8680" member_type: VOTER last_known_addr { host: "127.21.12.188" port: 35289 } attrs { promote: false } }
I20260806 06:26:51.296751 32263 leader_election.cc:288] T 00000000000000000000000000000000 P c88b3807d6bd413bb876555a253b8680 [CANDIDATE]: Term 2 election: Requested vote from peers 2589a7a56e404deda1060eb408ebf54c (127.21.12.190:41635), 124d842e9e484f8f85ddbc038dc51a31 (127.21.12.189:35823)
I20260806 06:26:51.296733 32264 raft_consensus.cc:504] T 00000000000000000000000000000000 P 124d842e9e484f8f85ddbc038dc51a31 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "2589a7a56e404deda1060eb408ebf54c" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 41635 } } peers { permanent_uuid: "124d842e9e484f8f85ddbc038dc51a31" member_type: VOTER last_known_addr { host: "127.21.12.189" port: 35823 } attrs { promote: false } } peers { permanent_uuid: "c88b3807d6bd413bb876555a253b8680" member_type: VOTER last_known_addr { host: "127.21.12.188" port: 35289 } attrs { promote: false } }
I20260806 06:26:51.296859 32264 leader_election.cc:288] T 00000000000000000000000000000000 P 124d842e9e484f8f85ddbc038dc51a31 [CANDIDATE]: Term 2 election: Requested vote from peers 2589a7a56e404deda1060eb408ebf54c (127.21.12.190:41635), c88b3807d6bd413bb876555a253b8680 (127.21.12.188:35289)
I20260806 06:26:51.297056 32270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:51.296977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:41635 (local address 127.0.0.1:51278)
0806 06:26:51.297006 (+ 29us) negotiation.cc:107] Waiting for socket to connect
0806 06:26:51.297032 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.190:41635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15}
I20260806 06:26:51.297144 31466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "124d842e9e484f8f85ddbc038dc51a31" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "c88b3807d6bd413bb876555a253b8680"
I20260806 06:26:51.297168 31181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c88b3807d6bd413bb876555a253b8680" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "124d842e9e484f8f85ddbc038dc51a31"
I20260806 06:26:51.297219 31466 raft_consensus.cc:2379] T 00000000000000000000000000000000 P c88b3807d6bd413bb876555a253b8680 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 124d842e9e484f8f85ddbc038dc51a31 in current term 2: Already voted for candidate c88b3807d6bd413bb876555a253b8680 in this term.
I20260806 06:26:51.297243 31181 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 124d842e9e484f8f85ddbc038dc51a31 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate c88b3807d6bd413bb876555a253b8680 in current term 2: Already voted for candidate 124d842e9e484f8f85ddbc038dc51a31 in this term.
I20260806 06:26:51.297603 32266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:51.296838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:41635 (local address 127.0.0.1:51262)
0806 06:26:51.297557 (+ 719us) negotiation.cc:107] Waiting for socket to connect
0806 06:26:51.297584 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.190:41635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":707}
I20260806 06:26:51.297691 32270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:51.297597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:41635 (local address 127.0.0.1:51296)
0806 06:26:51.297646 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0806 06:26:51.297672 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.190:41635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260806 06:26:51.297851 31150 leader_election.cc:334] T 00000000000000000000000000000000 P 124d842e9e484f8f85ddbc038dc51a31 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 2589a7a56e404deda1060eb408ebf54c (127.21.12.190:41635): Network error: Client connection negotiation failed: client connection to 127.21.12.190:41635: connect: Connection refused (error 111)
I20260806 06:26:51.297887 32266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:51.297815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:41635 (local address 127.0.0.1:51304)
0806 06:26:51.297841 (+ 26us) negotiation.cc:107] Waiting for socket to connect
0806 06:26:51.297864 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.190:41635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15}
I20260806 06:26:51.297905 31150 leader_election.cc:302] T 00000000000000000000000000000000 P 124d842e9e484f8f85ddbc038dc51a31 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 124d842e9e484f8f85ddbc038dc51a31; no voters: 2589a7a56e404deda1060eb408ebf54c, c88b3807d6bd413bb876555a253b8680
W20260806 06:26:51.297972 31434 leader_election.cc:334] T 00000000000000000000000000000000 P c88b3807d6bd413bb876555a253b8680 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 2589a7a56e404deda1060eb408ebf54c (127.21.12.190:41635): Network error: Client connection negotiation failed: client connection to 127.21.12.190:41635: connect: Connection refused (error 111)
I20260806 06:26:51.298013 31434 leader_election.cc:302] T 00000000000000000000000000000000 P c88b3807d6bd413bb876555a253b8680 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c88b3807d6bd413bb876555a253b8680; no voters: 124d842e9e484f8f85ddbc038dc51a31, 2589a7a56e404deda1060eb408ebf54c
I20260806 06:26:51.298082 32263 raft_consensus.cc:2726] T 00000000000000000000000000000000 P c88b3807d6bd413bb876555a253b8680 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260806 06:26:51.298110 32264 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 124d842e9e484f8f85ddbc038dc51a31 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260806 06:26:51.788069 32277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:51.787702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:41635 (local address 127.21.12.129:55617)
0806 06:26:51.787897 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0806 06:26:51.788011 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.190:41635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":86,"threads_started":1}
I20260806 06:26:51.788327 32278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:51.788097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:41635 (local address 127.21.12.131:46271)
0806 06:26:51.788249 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0806 06:26:51.788283 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.190:41635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":53,"threads_started":1}
I20260806 06:26:51.788481 32279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:51.788152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:41635 (local address 127.21.12.130:36651)
0806 06:26:51.788352 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0806 06:26:51.788440 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.190:41635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":71,"threads_started":1}
I20260806 06:26:52.788903 32281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:52.788693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:41635 (local address 127.21.12.131:51439)
0806 06:26:52.788805 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0806 06:26:52.788848 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.190:41635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":64,"threads_started":1}
I20260806 06:26:52.789108 32282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:52.788898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:41635 (local address 127.21.12.130:57069)
0806 06:26:52.789024 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0806 06:26:52.789062 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.190:41635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":44,"threads_started":1}
I20260806 06:26:52.789224 32280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:52.788661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:41635 (local address 127.21.12.129:59487)
0806 06:26:52.789147 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0806 06:26:52.789184 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.190:41635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":63,"threads_started":1}
I20260806 06:26:53.260540 32285 raft_consensus.cc:482] T 00000000000000000000000000000000 P c88b3807d6bd413bb876555a253b8680 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 06:26:53.260632 32285 raft_consensus.cc:504] T 00000000000000000000000000000000 P c88b3807d6bd413bb876555a253b8680 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "2589a7a56e404deda1060eb408ebf54c" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 41635 } } peers { permanent_uuid: "124d842e9e484f8f85ddbc038dc51a31" member_type: VOTER last_known_addr { host: "127.21.12.189" port: 35823 } attrs { promote: false } } peers { permanent_uuid: "c88b3807d6bd413bb876555a253b8680" member_type: VOTER last_known_addr { host: "127.21.12.188" port: 35289 } attrs { promote: false } }
I20260806 06:26:53.260802 32285 leader_election.cc:288] T 00000000000000000000000000000000 P c88b3807d6bd413bb876555a253b8680 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 2589a7a56e404deda1060eb408ebf54c (127.21.12.190:41635), 124d842e9e484f8f85ddbc038dc51a31 (127.21.12.189:35823)
I20260806 06:26:53.261066 31181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c88b3807d6bd413bb876555a253b8680" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "124d842e9e484f8f85ddbc038dc51a31" is_pre_election: true
I20260806 06:26:53.261123 32286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:53.260920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:41635 (local address 127.0.0.1:51534)
0806 06:26:53.261039 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0806 06:26:53.261084 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.190:41635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":68,"threads_started":1}
I20260806 06:26:53.261162 31181 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 124d842e9e484f8f85ddbc038dc51a31 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c88b3807d6bd413bb876555a253b8680 in term 2.
I20260806 06:26:53.261325 31433 leader_election.cc:302] T 00000000000000000000000000000000 P c88b3807d6bd413bb876555a253b8680 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 124d842e9e484f8f85ddbc038dc51a31, c88b3807d6bd413bb876555a253b8680; no voters:
I20260806 06:26:53.261421 32285 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c88b3807d6bd413bb876555a253b8680 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 06:26:53.261472 32285 raft_consensus.cc:482] T 00000000000000000000000000000000 P c88b3807d6bd413bb876555a253b8680 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 06:26:53.261492 32285 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c88b3807d6bd413bb876555a253b8680 [term 2 FOLLOWER]: Advancing to term 3
I20260806 06:26:53.261534 32286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:53.261420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:41635 (local address 127.0.0.1:51546)
0806 06:26:53.261456 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0806 06:26:53.261492 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.190:41635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
W20260806 06:26:53.261596 31434 leader_election.cc:334] T 00000000000000000000000000000000 P c88b3807d6bd413bb876555a253b8680 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 2589a7a56e404deda1060eb408ebf54c (127.21.12.190:41635): Network error: Client connection negotiation failed: client connection to 127.21.12.190:41635: connect: Connection refused (error 111)
I20260806 06:26:53.262249 32285 raft_consensus.cc:504] T 00000000000000000000000000000000 P c88b3807d6bd413bb876555a253b8680 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "2589a7a56e404deda1060eb408ebf54c" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 41635 } } peers { permanent_uuid: "124d842e9e484f8f85ddbc038dc51a31" member_type: VOTER last_known_addr { host: "127.21.12.189" port: 35823 } attrs { promote: false } } peers { permanent_uuid: "c88b3807d6bd413bb876555a253b8680" member_type: VOTER last_known_addr { host: "127.21.12.188" port: 35289 } attrs { promote: false } }
I20260806 06:26:53.262410 32285 leader_election.cc:288] T 00000000000000000000000000000000 P c88b3807d6bd413bb876555a253b8680 [CANDIDATE]: Term 3 election: Requested vote from peers 2589a7a56e404deda1060eb408ebf54c (127.21.12.190:41635), 124d842e9e484f8f85ddbc038dc51a31 (127.21.12.189:35823)
I20260806 06:26:53.262574 32286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:53.262495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:41635 (local address 127.0.0.1:51558)
0806 06:26:53.262527 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0806 06:26:53.262552 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.190:41635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
I20260806 06:26:53.262607 31181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c88b3807d6bd413bb876555a253b8680" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "124d842e9e484f8f85ddbc038dc51a31"
I20260806 06:26:53.262698 31181 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 124d842e9e484f8f85ddbc038dc51a31 [term 2 FOLLOWER]: Advancing to term 3
I20260806 06:26:53.262864 32286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:53.262795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:41635 (local address 127.0.0.1:51572)
0806 06:26:53.262823 (+ 28us) negotiation.cc:107] Waiting for socket to connect
0806 06:26:53.262846 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.190:41635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15}
W20260806 06:26:53.262945 31434 leader_election.cc:334] T 00000000000000000000000000000000 P c88b3807d6bd413bb876555a253b8680 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 2589a7a56e404deda1060eb408ebf54c (127.21.12.190:41635): Network error: Client connection negotiation failed: client connection to 127.21.12.190:41635: connect: Connection refused (error 111)
I20260806 06:26:53.263383 31181 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 124d842e9e484f8f85ddbc038dc51a31 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c88b3807d6bd413bb876555a253b8680 in term 3.
I20260806 06:26:53.263566 31433 leader_election.cc:302] T 00000000000000000000000000000000 P c88b3807d6bd413bb876555a253b8680 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 124d842e9e484f8f85ddbc038dc51a31, c88b3807d6bd413bb876555a253b8680; no voters: 2589a7a56e404deda1060eb408ebf54c
I20260806 06:26:53.263672 32285 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c88b3807d6bd413bb876555a253b8680 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 06:26:53.263790 32285 raft_consensus.cc:686] T 00000000000000000000000000000000 P c88b3807d6bd413bb876555a253b8680 [term 3 LEADER]: Becoming Leader. State: Replica: c88b3807d6bd413bb876555a253b8680, State: Running, Role: LEADER
I20260806 06:26:53.263871 32285 consensus_queue.cc:237] T 00000000000000000000000000000000 P c88b3807d6bd413bb876555a253b8680 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "2589a7a56e404deda1060eb408ebf54c" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 41635 } } peers { permanent_uuid: "124d842e9e484f8f85ddbc038dc51a31" member_type: VOTER last_known_addr { host: "127.21.12.189" port: 35823 } attrs { promote: false } } peers { permanent_uuid: "c88b3807d6bd413bb876555a253b8680" member_type: VOTER last_known_addr { host: "127.21.12.188" port: 35289 } attrs { promote: false } }
I20260806 06:26:53.264247 32289 sys_catalog.cc:455] T 00000000000000000000000000000000 P c88b3807d6bd413bb876555a253b8680 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c88b3807d6bd413bb876555a253b8680. Latest consensus state: current_term: 3 leader_uuid: "c88b3807d6bd413bb876555a253b8680" committed_config { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "2589a7a56e404deda1060eb408ebf54c" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 41635 } } peers { permanent_uuid: "124d842e9e484f8f85ddbc038dc51a31" member_type: VOTER last_known_addr { host: "127.21.12.189" port: 35823 } attrs { promote: false } } peers { permanent_uuid: "c88b3807d6bd413bb876555a253b8680" member_type: VOTER last_known_addr { host: "127.21.12.188" port: 35289 } attrs { promote: false } } }
I20260806 06:26:53.264335 32289 sys_catalog.cc:458] T 00000000000000000000000000000000 P c88b3807d6bd413bb876555a253b8680 [sys.catalog]: This master's current role is: LEADER
I20260806 06:26:53.264516 32291 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 06:26:53.265044 32291 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 06:26:53.265122 32291 catalog_manager.cc:1259] Loaded cluster ID: 2793aaf3eaab49c4806dfa641e5153fa
I20260806 06:26:53.265154 32291 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 06:26:53.265347 32291 catalog_manager.cc:1504] Loading token signing keys...
I20260806 06:26:53.265415 32291 catalog_manager.cc:5879] T 00000000000000000000000000000000 P c88b3807d6bd413bb876555a253b8680: Loaded TSK: 0
I20260806 06:26:53.265585 32291 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 06:26:53.301290 31452 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35354:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 06:26:53.303004 31452 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 06:26:53.303997 32286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:53.303892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:41635 (local address 127.0.0.1:51600)
0806 06:26:53.303928 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0806 06:26:53.303964 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.190:41635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
I20260806 06:26:53.304016 31181 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 124d842e9e484f8f85ddbc038dc51a31 [term 3 FOLLOWER]: Refusing update from remote peer c88b3807d6bd413bb876555a253b8680: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260806 06:26:53.304219 32285 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c88b3807d6bd413bb876555a253b8680 [LEADER]: Connected to new peer: Peer: permanent_uuid: "124d842e9e484f8f85ddbc038dc51a31" member_type: VOTER last_known_addr { host: "127.21.12.189" port: 35823 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260806 06:26:53.304360 32286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:53.304262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:41635 (local address 127.0.0.1:51606)
0806 06:26:53.304302 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0806 06:26:53.304331 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.190:41635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260806 06:26:53.304440 31434 consensus_peers.cc:487] T 00000000000000000000000000000000 P c88b3807d6bd413bb876555a253b8680 -> Peer 2589a7a56e404deda1060eb408ebf54c (127.21.12.190:41635): Couldn't send request to peer 2589a7a56e404deda1060eb408ebf54c. Status: Network error: Client connection negotiation failed: client connection to 127.21.12.190:41635: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260806 06:26:53.304819 32293 sys_catalog.cc:455] T 00000000000000000000000000000000 P 124d842e9e484f8f85ddbc038dc51a31 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c88b3807d6bd413bb876555a253b8680. Latest consensus state: current_term: 3 leader_uuid: "c88b3807d6bd413bb876555a253b8680" committed_config { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "2589a7a56e404deda1060eb408ebf54c" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 41635 } } peers { permanent_uuid: "124d842e9e484f8f85ddbc038dc51a31" member_type: VOTER last_known_addr { host: "127.21.12.189" port: 35823 } attrs { promote: false } } peers { permanent_uuid: "c88b3807d6bd413bb876555a253b8680" member_type: VOTER last_known_addr { host: "127.21.12.188" port: 35289 } attrs { promote: false } } }
I20260806 06:26:53.304982 32293 sys_catalog.cc:458] T 00000000000000000000000000000000 P 124d842e9e484f8f85ddbc038dc51a31 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 06:26:53.305373 32289 sys_catalog.cc:455] T 00000000000000000000000000000000 P c88b3807d6bd413bb876555a253b8680 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "c88b3807d6bd413bb876555a253b8680" committed_config { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "2589a7a56e404deda1060eb408ebf54c" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 41635 } } peers { permanent_uuid: "124d842e9e484f8f85ddbc038dc51a31" member_type: VOTER last_known_addr { host: "127.21.12.189" port: 35823 } attrs { promote: false } } peers { permanent_uuid: "c88b3807d6bd413bb876555a253b8680" member_type: VOTER last_known_addr { host: "127.21.12.188" port: 35289 } attrs { promote: false } } }
I20260806 06:26:53.305459 32289 sys_catalog.cc:458] T 00000000000000000000000000000000 P c88b3807d6bd413bb876555a253b8680 [sys.catalog]: This master's current role is: LEADER
I20260806 06:26:53.305591 32293 sys_catalog.cc:455] T 00000000000000000000000000000000 P 124d842e9e484f8f85ddbc038dc51a31 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "c88b3807d6bd413bb876555a253b8680" committed_config { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "2589a7a56e404deda1060eb408ebf54c" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 41635 } } peers { permanent_uuid: "124d842e9e484f8f85ddbc038dc51a31" member_type: VOTER last_known_addr { host: "127.21.12.189" port: 35823 } attrs { promote: false } } peers { permanent_uuid: "c88b3807d6bd413bb876555a253b8680" member_type: VOTER last_known_addr { host: "127.21.12.188" port: 35289 } attrs { promote: false } } }
I20260806 06:26:53.305665 32293 sys_catalog.cc:458] T 00000000000000000000000000000000 P 124d842e9e484f8f85ddbc038dc51a31 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 06:26:53.310648 32310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:53.307329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52126 (local address 127.21.12.129:41895)
0806 06:26:53.307424 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0806 06:26:53.307426 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:26:53.307434 (+ 8us) server_negotiation.cc:407] Connection header received
0806 06:26:53.307459 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:26:53.307460 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:26:53.307470 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:26:53.307865 (+ 395us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 06:26:53.307907 (+ 42us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:53.308622 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:53.309810 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:53.310318 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:53.310421 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:53.310435 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0806 06:26:53.310535 (+ 100us) server_negotiation.cc:299] Negotiation successful
0806 06:26:53.310575 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77,"thread_start_us":50,"threads_started":1}
I20260806 06:26:53.310664 32300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:53.307096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.129:41895 (local address 127.0.0.1:52126)
0806 06:26:53.307200 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0806 06:26:53.307209 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0806 06:26:53.307245 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:26:53.307748 (+ 503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:26:53.307749 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:26:53.307751 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 06:26:53.307804 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:53.307806 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:53.308696 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:53.308697 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:53.309752 (+ 1055us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:53.309753 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:53.310385 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:53.310386 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:53.310448 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:53.310452 (+ 4us) client_negotiation.cc:769] Sending connection context
0806 06:26:53.310504 (+ 52us) client_negotiation.cc:240] Negotiation successful
0806 06:26:53.310548 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":88,"thread_start_us":71,"threads_started":1}
I20260806 06:26:53.310835 32305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:53.307213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51982 (local address 127.21.12.130:45263)
0806 06:26:53.307581 (+ 368us) server_negotiation.cc:206] Beginning negotiation
0806 06:26:53.307584 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 06:26:53.307591 (+ 7us) server_negotiation.cc:407] Connection header received
0806 06:26:53.307616 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:26:53.307616 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:26:53.307626 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:26:53.307682 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 06:26:53.307814 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:53.308301 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:53.309669 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:53.310323 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:53.310690 (+ 367us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:53.310708 (+ 18us) server_negotiation.cc:1035] Waiting for connection context
0806 06:26:53.310728 (+ 20us) server_negotiation.cc:299] Negotiation successful
0806 06:26:53.310766 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":346,"thread_start_us":46,"threads_started":1}
I20260806 06:26:53.310675 32302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:53.307131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.130:45263 (local address 127.0.0.1:51982)
0806 06:26:53.307315 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0806 06:26:53.307328 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 06:26:53.307380 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:26:53.307695 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:26:53.307696 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:26:53.307698 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 06:26:53.307730 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:53.307731 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:53.308379 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:53.308380 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:53.309576 (+ 1196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:53.309579 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:53.310432 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:53.310433 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:53.310495 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:53.310500 (+ 5us) client_negotiation.cc:769] Sending connection context
0806 06:26:53.310535 (+ 35us) client_negotiation.cc:240] Negotiation successful
0806 06:26:53.310577 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"mutex_wait_us":31,"thread_start_us":98,"threads_started":1}
I20260806 06:26:53.311148 31862 tablet_service.cc:1467] Processing CreateTablet for tablet 31e6065acdb644049bf164c8c12861d8 (DEFAULT_TABLE table=test-table [id=c9ab14ad6c09431d9f427a817fab667f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 06:26:53.311151 31637 tablet_service.cc:1467] Processing CreateTablet for tablet 31e6065acdb644049bf164c8c12861d8 (DEFAULT_TABLE table=test-table [id=c9ab14ad6c09431d9f427a817fab667f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 06:26:53.311282 31637 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31e6065acdb644049bf164c8c12861d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 06:26:53.311297 31862 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31e6065acdb644049bf164c8c12861d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 06:26:53.311612 32286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:53.307019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.131:46131 (local address 127.0.0.1:46970)
0806 06:26:53.307057 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0806 06:26:53.307066 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0806 06:26:53.307111 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:26:53.307571 (+ 460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:26:53.307571 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:26:53.307573 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 06:26:53.307625 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:53.307627 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:53.308502 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:53.308503 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:53.310655 (+ 2152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:53.310657 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:53.311372 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:53.311398 (+ 26us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:53.311459 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:53.311463 (+ 4us) client_negotiation.cc:769] Sending connection context
0806 06:26:53.311511 (+ 48us) client_negotiation.cc:240] Negotiation successful
0806 06:26:53.311547 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24,"mutex_wait_us":1025}
I20260806 06:26:53.311673 32308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:53.307267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46970 (local address 127.21.12.131:46131)
0806 06:26:53.307425 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0806 06:26:53.307427 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:26:53.307434 (+ 7us) server_negotiation.cc:407] Connection header received
0806 06:26:53.307459 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:26:53.307460 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:26:53.307470 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:26:53.307570 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 06:26:53.307674 (+ 104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:53.308413 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:53.310743 (+ 2330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:53.311283 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:53.311351 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:53.311364 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0806 06:26:53.311519 (+ 155us) server_negotiation.cc:299] Negotiation successful
0806 06:26:53.311606 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":103,"threads_started":1}
I20260806 06:26:53.312608 32313 tablet_bootstrap.cc:492] T 31e6065acdb644049bf164c8c12861d8 P 9c272304769843f1af8f68c3d16ec185: Bootstrap starting.
I20260806 06:26:53.312803 32055 tablet_service.cc:1467] Processing CreateTablet for tablet 31e6065acdb644049bf164c8c12861d8 (DEFAULT_TABLE table=test-table [id=c9ab14ad6c09431d9f427a817fab667f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 06:26:53.312945 32055 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31e6065acdb644049bf164c8c12861d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 06:26:53.313022 32313 tablet_bootstrap.cc:654] T 31e6065acdb644049bf164c8c12861d8 P 9c272304769843f1af8f68c3d16ec185: Neither blocks nor log segments found. Creating new log.
I20260806 06:26:53.313253 32312 tablet_bootstrap.cc:492] T 31e6065acdb644049bf164c8c12861d8 P 08f0835258af4cfc8d2b541884f9ceba: Bootstrap starting.
I20260806 06:26:53.313587 32312 tablet_bootstrap.cc:654] T 31e6065acdb644049bf164c8c12861d8 P 08f0835258af4cfc8d2b541884f9ceba: Neither blocks nor log segments found. Creating new log.
I20260806 06:26:53.313933 32313 tablet_bootstrap.cc:492] T 31e6065acdb644049bf164c8c12861d8 P 9c272304769843f1af8f68c3d16ec185: No bootstrap required, opened a new log
I20260806 06:26:53.313977 32313 ts_tablet_manager.cc:1397] T 31e6065acdb644049bf164c8c12861d8 P 9c272304769843f1af8f68c3d16ec185: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260806 06:26:53.314109 32313 raft_consensus.cc:348] T 31e6065acdb644049bf164c8c12861d8 P 9c272304769843f1af8f68c3d16ec185 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79d4ece6336b48e59c3e65c1d11c3ded" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 46131 } } peers { permanent_uuid: "9c272304769843f1af8f68c3d16ec185" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 41895 } } peers { permanent_uuid: "08f0835258af4cfc8d2b541884f9ceba" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 45263 } }
I20260806 06:26:53.314183 32313 raft_consensus.cc:374] T 31e6065acdb644049bf164c8c12861d8 P 9c272304769843f1af8f68c3d16ec185 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 06:26:53.314203 32313 raft_consensus.cc:729] T 31e6065acdb644049bf164c8c12861d8 P 9c272304769843f1af8f68c3d16ec185 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c272304769843f1af8f68c3d16ec185, State: Initialized, Role: FOLLOWER
I20260806 06:26:53.314227 32312 tablet_bootstrap.cc:492] T 31e6065acdb644049bf164c8c12861d8 P 08f0835258af4cfc8d2b541884f9ceba: No bootstrap required, opened a new log
I20260806 06:26:53.314265 32312 ts_tablet_manager.cc:1397] T 31e6065acdb644049bf164c8c12861d8 P 08f0835258af4cfc8d2b541884f9ceba: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260806 06:26:53.314249 32313 consensus_queue.cc:260] T 31e6065acdb644049bf164c8c12861d8 P 9c272304769843f1af8f68c3d16ec185 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79d4ece6336b48e59c3e65c1d11c3ded" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 46131 } } peers { permanent_uuid: "9c272304769843f1af8f68c3d16ec185" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 41895 } } peers { permanent_uuid: "08f0835258af4cfc8d2b541884f9ceba" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 45263 } }
I20260806 06:26:53.314422 32313 ts_tablet_manager.cc:1428] T 31e6065acdb644049bf164c8c12861d8 P 9c272304769843f1af8f68c3d16ec185: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260806 06:26:53.314410 32312 raft_consensus.cc:348] T 31e6065acdb644049bf164c8c12861d8 P 08f0835258af4cfc8d2b541884f9ceba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79d4ece6336b48e59c3e65c1d11c3ded" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 46131 } } peers { permanent_uuid: "9c272304769843f1af8f68c3d16ec185" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 41895 } } peers { permanent_uuid: "08f0835258af4cfc8d2b541884f9ceba" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 45263 } }
I20260806 06:26:53.314471 32312 raft_consensus.cc:374] T 31e6065acdb644049bf164c8c12861d8 P 08f0835258af4cfc8d2b541884f9ceba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 06:26:53.314433 32320 tablet_bootstrap.cc:492] T 31e6065acdb644049bf164c8c12861d8 P 79d4ece6336b48e59c3e65c1d11c3ded: Bootstrap starting.
I20260806 06:26:53.314498 32312 raft_consensus.cc:729] T 31e6065acdb644049bf164c8c12861d8 P 08f0835258af4cfc8d2b541884f9ceba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08f0835258af4cfc8d2b541884f9ceba, State: Initialized, Role: FOLLOWER
I20260806 06:26:53.314551 32312 consensus_queue.cc:260] T 31e6065acdb644049bf164c8c12861d8 P 08f0835258af4cfc8d2b541884f9ceba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79d4ece6336b48e59c3e65c1d11c3ded" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 46131 } } peers { permanent_uuid: "9c272304769843f1af8f68c3d16ec185" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 41895 } } peers { permanent_uuid: "08f0835258af4cfc8d2b541884f9ceba" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 45263 } }
I20260806 06:26:53.314699 32312 ts_tablet_manager.cc:1428] T 31e6065acdb644049bf164c8c12861d8 P 08f0835258af4cfc8d2b541884f9ceba: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260806 06:26:53.314817 32320 tablet_bootstrap.cc:654] T 31e6065acdb644049bf164c8c12861d8 P 79d4ece6336b48e59c3e65c1d11c3ded: Neither blocks nor log segments found. Creating new log.
I20260806 06:26:53.315303 32320 tablet_bootstrap.cc:492] T 31e6065acdb644049bf164c8c12861d8 P 79d4ece6336b48e59c3e65c1d11c3ded: No bootstrap required, opened a new log
I20260806 06:26:53.315351 32320 ts_tablet_manager.cc:1397] T 31e6065acdb644049bf164c8c12861d8 P 79d4ece6336b48e59c3e65c1d11c3ded: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260806 06:26:53.315364 31754 heartbeater.cc:503] Master 127.21.12.188:35289 was elected leader, sending a full tablet report...
I20260806 06:26:53.315387 32326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:53.314945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:41635 (local address 127.21.12.130:57973)
0806 06:26:53.315308 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0806 06:26:53.315355 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.190:41635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":72,"threads_started":1}
I20260806 06:26:53.315487 32320 raft_consensus.cc:348] T 31e6065acdb644049bf164c8c12861d8 P 79d4ece6336b48e59c3e65c1d11c3ded [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79d4ece6336b48e59c3e65c1d11c3ded" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 46131 } } peers { permanent_uuid: "9c272304769843f1af8f68c3d16ec185" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 41895 } } peers { permanent_uuid: "08f0835258af4cfc8d2b541884f9ceba" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 45263 } }
I20260806 06:26:53.315560 32320 raft_consensus.cc:374] T 31e6065acdb644049bf164c8c12861d8 P 79d4ece6336b48e59c3e65c1d11c3ded [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 06:26:53.315582 32320 raft_consensus.cc:729] T 31e6065acdb644049bf164c8c12861d8 P 79d4ece6336b48e59c3e65c1d11c3ded [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79d4ece6336b48e59c3e65c1d11c3ded, State: Initialized, Role: FOLLOWER
I20260806 06:26:53.315619 32320 consensus_queue.cc:260] T 31e6065acdb644049bf164c8c12861d8 P 79d4ece6336b48e59c3e65c1d11c3ded [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79d4ece6336b48e59c3e65c1d11c3ded" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 46131 } } peers { permanent_uuid: "9c272304769843f1af8f68c3d16ec185" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 41895 } } peers { permanent_uuid: "08f0835258af4cfc8d2b541884f9ceba" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 45263 } }
I20260806 06:26:53.315758 32320 ts_tablet_manager.cc:1428] T 31e6065acdb644049bf164c8c12861d8 P 79d4ece6336b48e59c3e65c1d11c3ded: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260806 06:26:53.316021 32324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:53.314731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:41635 (local address 127.21.12.129:56163)
0806 06:26:53.315942 (+ 1211us) negotiation.cc:107] Waiting for socket to connect
0806 06:26:53.315980 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.190:41635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1182,"thread_start_us":69,"threads_started":1}
I20260806 06:26:53.316341 32324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:53.316182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:41635 (local address 127.21.12.129:46341)
0806 06:26:53.316293 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0806 06:26:53.316317 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.190:41635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":77,"threads_started":1}
I20260806 06:26:53.316331 32153 heartbeater.cc:503] Master 127.21.12.188:35289 was elected leader, sending a full tablet report...
I20260806 06:26:53.316382 32329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:53.316175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:41635 (local address 127.21.12.131:48319)
0806 06:26:53.316305 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0806 06:26:53.316345 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.190:41635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":79,"threads_started":1}
I20260806 06:26:53.316499 31949 heartbeater.cc:503] Master 127.21.12.188:35289 was elected leader, sending a full tablet report...
I20260806 06:26:53.316854 32326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:53.316670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:41635 (local address 127.21.12.130:44419)
0806 06:26:53.316802 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0806 06:26:53.316830 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.190:41635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117}
W20260806 06:26:53.325106 32156 tablet.cc:2369] T 31e6065acdb644049bf164c8c12861d8 P 79d4ece6336b48e59c3e65c1d11c3ded: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 06:26:53.468509 32327 raft_consensus.cc:482] T 31e6065acdb644049bf164c8c12861d8 P 79d4ece6336b48e59c3e65c1d11c3ded [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 06:26:53.468621 32327 raft_consensus.cc:504] T 31e6065acdb644049bf164c8c12861d8 P 79d4ece6336b48e59c3e65c1d11c3ded [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79d4ece6336b48e59c3e65c1d11c3ded" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 46131 } } peers { permanent_uuid: "9c272304769843f1af8f68c3d16ec185" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 41895 } } peers { permanent_uuid: "08f0835258af4cfc8d2b541884f9ceba" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 45263 } }
I20260806 06:26:53.468806 32327 leader_election.cc:288] T 31e6065acdb644049bf164c8c12861d8 P 79d4ece6336b48e59c3e65c1d11c3ded [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9c272304769843f1af8f68c3d16ec185 (127.21.12.129:41895), 08f0835258af4cfc8d2b541884f9ceba (127.21.12.130:45263)
I20260806 06:26:53.469033 31679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31e6065acdb644049bf164c8c12861d8" candidate_uuid: "79d4ece6336b48e59c3e65c1d11c3ded" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c272304769843f1af8f68c3d16ec185" is_pre_election: true
I20260806 06:26:53.469048 31887 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31e6065acdb644049bf164c8c12861d8" candidate_uuid: "79d4ece6336b48e59c3e65c1d11c3ded" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08f0835258af4cfc8d2b541884f9ceba" is_pre_election: true
I20260806 06:26:53.469139 31679 raft_consensus.cc:2454] T 31e6065acdb644049bf164c8c12861d8 P 9c272304769843f1af8f68c3d16ec185 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 79d4ece6336b48e59c3e65c1d11c3ded in term 0.
I20260806 06:26:53.469139 31887 raft_consensus.cc:2454] T 31e6065acdb644049bf164c8c12861d8 P 08f0835258af4cfc8d2b541884f9ceba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 79d4ece6336b48e59c3e65c1d11c3ded in term 0.
I20260806 06:26:53.469307 31981 leader_election.cc:302] T 31e6065acdb644049bf164c8c12861d8 P 79d4ece6336b48e59c3e65c1d11c3ded [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08f0835258af4cfc8d2b541884f9ceba, 79d4ece6336b48e59c3e65c1d11c3ded; no voters:
I20260806 06:26:53.469416 32327 raft_consensus.cc:2781] T 31e6065acdb644049bf164c8c12861d8 P 79d4ece6336b48e59c3e65c1d11c3ded [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 06:26:53.469482 32327 raft_consensus.cc:482] T 31e6065acdb644049bf164c8c12861d8 P 79d4ece6336b48e59c3e65c1d11c3ded [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 06:26:53.469497 32327 raft_consensus.cc:3037] T 31e6065acdb644049bf164c8c12861d8 P 79d4ece6336b48e59c3e65c1d11c3ded [term 0 FOLLOWER]: Advancing to term 1
I20260806 06:26:53.470176 32327 raft_consensus.cc:504] T 31e6065acdb644049bf164c8c12861d8 P 79d4ece6336b48e59c3e65c1d11c3ded [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79d4ece6336b48e59c3e65c1d11c3ded" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 46131 } } peers { permanent_uuid: "9c272304769843f1af8f68c3d16ec185" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 41895 } } peers { permanent_uuid: "08f0835258af4cfc8d2b541884f9ceba" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 45263 } }
I20260806 06:26:53.470314 32327 leader_election.cc:288] T 31e6065acdb644049bf164c8c12861d8 P 79d4ece6336b48e59c3e65c1d11c3ded [CANDIDATE]: Term 1 election: Requested vote from peers 9c272304769843f1af8f68c3d16ec185 (127.21.12.129:41895), 08f0835258af4cfc8d2b541884f9ceba (127.21.12.130:45263)
I20260806 06:26:53.470479 31887 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31e6065acdb644049bf164c8c12861d8" candidate_uuid: "79d4ece6336b48e59c3e65c1d11c3ded" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08f0835258af4cfc8d2b541884f9ceba"
I20260806 06:26:53.470495 31679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31e6065acdb644049bf164c8c12861d8" candidate_uuid: "79d4ece6336b48e59c3e65c1d11c3ded" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c272304769843f1af8f68c3d16ec185"
I20260806 06:26:53.470544 31887 raft_consensus.cc:3037] T 31e6065acdb644049bf164c8c12861d8 P 08f0835258af4cfc8d2b541884f9ceba [term 0 FOLLOWER]: Advancing to term 1
I20260806 06:26:53.470564 31679 raft_consensus.cc:3037] T 31e6065acdb644049bf164c8c12861d8 P 9c272304769843f1af8f68c3d16ec185 [term 0 FOLLOWER]: Advancing to term 1
I20260806 06:26:53.471153 31887 raft_consensus.cc:2454] T 31e6065acdb644049bf164c8c12861d8 P 08f0835258af4cfc8d2b541884f9ceba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 79d4ece6336b48e59c3e65c1d11c3ded in term 1.
I20260806 06:26:53.471160 31679 raft_consensus.cc:2454] T 31e6065acdb644049bf164c8c12861d8 P 9c272304769843f1af8f68c3d16ec185 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 79d4ece6336b48e59c3e65c1d11c3ded in term 1.
I20260806 06:26:53.471315 31981 leader_election.cc:302] T 31e6065acdb644049bf164c8c12861d8 P 79d4ece6336b48e59c3e65c1d11c3ded [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08f0835258af4cfc8d2b541884f9ceba, 79d4ece6336b48e59c3e65c1d11c3ded; no voters:
I20260806 06:26:53.471416 32327 raft_consensus.cc:2781] T 31e6065acdb644049bf164c8c12861d8 P 79d4ece6336b48e59c3e65c1d11c3ded [term 1 FOLLOWER]: Leader election won for term 1
I20260806 06:26:53.471580 32327 raft_consensus.cc:686] T 31e6065acdb644049bf164c8c12861d8 P 79d4ece6336b48e59c3e65c1d11c3ded [term 1 LEADER]: Becoming Leader. State: Replica: 79d4ece6336b48e59c3e65c1d11c3ded, State: Running, Role: LEADER
I20260806 06:26:53.471629 32327 consensus_queue.cc:237] T 31e6065acdb644049bf164c8c12861d8 P 79d4ece6336b48e59c3e65c1d11c3ded [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79d4ece6336b48e59c3e65c1d11c3ded" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 46131 } } peers { permanent_uuid: "9c272304769843f1af8f68c3d16ec185" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 41895 } } peers { permanent_uuid: "08f0835258af4cfc8d2b541884f9ceba" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 45263 } }
I20260806 06:26:53.471936 32329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:53.471821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:41635 (local address 127.21.12.131:43823)
0806 06:26:53.471873 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0806 06:26:53.471907 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.190:41635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260806 06:26:53.472234 31452 catalog_manager.cc:5495] T 31e6065acdb644049bf164c8c12861d8 P 79d4ece6336b48e59c3e65c1d11c3ded reported cstate change: term changed from 0 to 1, leader changed from <none> to 79d4ece6336b48e59c3e65c1d11c3ded (127.21.12.131). New cstate: current_term: 1 leader_uuid: "79d4ece6336b48e59c3e65c1d11c3ded" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79d4ece6336b48e59c3e65c1d11c3ded" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 46131 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c272304769843f1af8f68c3d16ec185" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 41895 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "08f0835258af4cfc8d2b541884f9ceba" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 45263 } health_report { overall_health: UNKNOWN } } }
I20260806 06:26:53.491535 32308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:53.489470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46972 (local address 127.21.12.131:46131)
0806 06:26:53.489523 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0806 06:26:53.489525 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:26:53.489536 (+ 11us) server_negotiation.cc:407] Connection header received
0806 06:26:53.489562 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:26:53.489563 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:26:53.489572 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:26:53.489629 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 06:26:53.489732 (+ 103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:53.490218 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:53.490983 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:53.491197 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:53.491255 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:53.491353 (+ 98us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 06:26:53.491380 (+ 27us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 06:26:53.491415 (+ 35us) server_negotiation.cc:1035] Waiting for connection context
0806 06:26:53.491450 (+ 35us) server_negotiation.cc:299] Negotiation successful
0806 06:26:53.491469 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260806 06:26:53.492406 31679 raft_consensus.cc:1261] T 31e6065acdb644049bf164c8c12861d8 P 9c272304769843f1af8f68c3d16ec185 [term 1 FOLLOWER]: Refusing update from remote peer 79d4ece6336b48e59c3e65c1d11c3ded: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 06:26:53.492434 31887 raft_consensus.cc:1261] T 31e6065acdb644049bf164c8c12861d8 P 08f0835258af4cfc8d2b541884f9ceba [term 1 FOLLOWER]: Refusing update from remote peer 79d4ece6336b48e59c3e65c1d11c3ded: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 06:26:53.492688 32327 consensus_queue.cc:1035] T 31e6065acdb644049bf164c8c12861d8 P 79d4ece6336b48e59c3e65c1d11c3ded [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c272304769843f1af8f68c3d16ec185" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 41895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 06:26:53.492806 32330 consensus_queue.cc:1035] T 31e6065acdb644049bf164c8c12861d8 P 79d4ece6336b48e59c3e65c1d11c3ded [LEADER]: Connected to new peer: Peer: permanent_uuid: "08f0835258af4cfc8d2b541884f9ceba" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 45263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 06:26:53.493523 32326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:53.493414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:41635 (local address 127.21.12.130:34161)
0806 06:26:53.493448 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0806 06:26:53.493490 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.190:41635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
I20260806 06:26:53.494032 32328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:53.493573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:41635 (local address 127.21.12.129:42145)
0806 06:26:53.493943 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0806 06:26:53.493984 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.190:41635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":343}
I20260806 06:26:53.494402 32335 mvcc.cc:205] Tried to move back new op lower bound from 7315446224863010816 to 7315446224780115968. Current Snapshot: MvccSnapshot[applied={T|T < 7315446224863010816}]
I20260806 06:26:53.494863 32329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:53.494713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:41635 (local address 127.21.12.131:37025)
0806 06:26:53.494804 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0806 06:26:53.494833 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.190:41635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260806 06:26:53.495307 32328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:53.495139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:41635 (local address 127.21.12.129:47795)
0806 06:26:53.495251 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0806 06:26:53.495277 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.190:41635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260806 06:26:53.495383 32329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:53.495242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:41635 (local address 127.21.12.131:44701)
0806 06:26:53.495341 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0806 06:26:53.495363 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.190:41635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260806 06:26:53.496316 32326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:53.496020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:41635 (local address 127.21.12.130:49891)
0806 06:26:53.496254 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0806 06:26:53.496287 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.190:41635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218}
I20260806 06:26:53.496443 32337 mvcc.cc:205] Tried to move back new op lower bound from 7315446224863010816 to 7315446224780115968. Current Snapshot: MvccSnapshot[applied={T|T < 7315446224863010816 or (T in {7315446224863010816})}]
I20260806 06:26:53.497084 31447 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35354:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 06:26:53.497164 31447 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35354:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 06:26:53.498365 31447 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c88b3807d6bd413bb876555a253b8680: Sending DeleteTablet for 3 replicas of tablet 31e6065acdb644049bf164c8c12861d8
I20260806 06:26:53.498641 32055 tablet_service.cc:1514] Processing DeleteTablet for tablet 31e6065acdb644049bf164c8c12861d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 06:26:53 UTC) from {username='oryx', principal='oryx/127.21.12.188@KRBTEST.COM'} at 127.0.0.1:46970
I20260806 06:26:53.498647 31637 tablet_service.cc:1514] Processing DeleteTablet for tablet 31e6065acdb644049bf164c8c12861d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 06:26:53 UTC) from {username='oryx', principal='oryx/127.21.12.188@KRBTEST.COM'} at 127.0.0.1:52126
I20260806 06:26:53.498672 31862 tablet_service.cc:1514] Processing DeleteTablet for tablet 31e6065acdb644049bf164c8c12861d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 06:26:53 UTC) from {username='oryx', principal='oryx/127.21.12.188@KRBTEST.COM'} at 127.0.0.1:51982
I20260806 06:26:53.498827 32349 tablet_replica.cc:330] T 31e6065acdb644049bf164c8c12861d8 P 79d4ece6336b48e59c3e65c1d11c3ded: stopping tablet replica
I20260806 06:26:53.498879 32349 raft_consensus.cc:2229] T 31e6065acdb644049bf164c8c12861d8 P 79d4ece6336b48e59c3e65c1d11c3ded [term 1 LEADER]: Raft consensus shutting down.
I20260806 06:26:53.498975 32349 raft_consensus.cc:2258] T 31e6065acdb644049bf164c8c12861d8 P 79d4ece6336b48e59c3e65c1d11c3ded [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 06:26:53.499042 21554 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHhLhT/build/release/bin/kudu with pid 31404
I20260806 06:26:53.515987 32350 tablet_replica.cc:330] T 31e6065acdb644049bf164c8c12861d8 P 08f0835258af4cfc8d2b541884f9ceba: stopping tablet replica
I20260806 06:26:53.516088 32350 raft_consensus.cc:2229] T 31e6065acdb644049bf164c8c12861d8 P 08f0835258af4cfc8d2b541884f9ceba [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 06:26:53.516170 32350 raft_consensus.cc:2258] T 31e6065acdb644049bf164c8c12861d8 P 08f0835258af4cfc8d2b541884f9ceba [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 06:26:53.516690 32349 ts_tablet_manager.cc:1905] T 31e6065acdb644049bf164c8c12861d8 P 79d4ece6336b48e59c3e65c1d11c3ded: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 06:26:53.517117 32350 ts_tablet_manager.cc:1905] T 31e6065acdb644049bf164c8c12861d8 P 08f0835258af4cfc8d2b541884f9ceba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 06:26:53.518463 32350 ts_tablet_manager.cc:1918] T 31e6065acdb644049bf164c8c12861d8 P 08f0835258af4cfc8d2b541884f9ceba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 06:26:53.518477 32349 ts_tablet_manager.cc:1918] T 31e6065acdb644049bf164c8c12861d8 P 79d4ece6336b48e59c3e65c1d11c3ded: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 06:26:53.518517 32350 log.cc:1196] T 31e6065acdb644049bf164c8c12861d8 P 08f0835258af4cfc8d2b541884f9ceba: Deleting WAL directory at /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785997581049705-21554-0/minicluster-data/ts-1/wal/wals/31e6065acdb644049bf164c8c12861d8
I20260806 06:26:53.518534 32349 log.cc:1196] T 31e6065acdb644049bf164c8c12861d8 P 79d4ece6336b48e59c3e65c1d11c3ded: Deleting WAL directory at /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785997581049705-21554-0/minicluster-data/ts-2/wal/wals/31e6065acdb644049bf164c8c12861d8
I20260806 06:26:53.518774 32350 ts_tablet_manager.cc:1939] T 31e6065acdb644049bf164c8c12861d8 P 08f0835258af4cfc8d2b541884f9ceba: Deleting consensus metadata
I20260806 06:26:53.518801 32349 ts_tablet_manager.cc:1939] T 31e6065acdb644049bf164c8c12861d8 P 79d4ece6336b48e59c3e65c1d11c3ded: Deleting consensus metadata
I20260806 06:26:53.519145 31432 catalog_manager.cc:4841] TS 08f0835258af4cfc8d2b541884f9ceba (127.21.12.130:45263): tablet 31e6065acdb644049bf164c8c12861d8 (table test-table [id=c9ab14ad6c09431d9f427a817fab667f]) successfully deleted
I20260806 06:26:53.519145 31430 catalog_manager.cc:4841] TS 79d4ece6336b48e59c3e65c1d11c3ded (127.21.12.131:46131): tablet 31e6065acdb644049bf164c8c12861d8 (table test-table [id=c9ab14ad6c09431d9f427a817fab667f]) successfully deleted
I20260806 06:26:53.520963 32286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:53.520849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.129:41895 (local address 127.0.0.1:52128)
0806 06:26:53.520901 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0806 06:26:53.520931 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.129:41895: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260806 06:26:53.521045 21554 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHhLhT/build/release/bin/kudu with pid 31764
W20260806 06:26:53.521149 31434 catalog_manager.cc:4584] TS 9c272304769843f1af8f68c3d16ec185 (127.21.12.129:41895): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 31e6065acdb644049bf164c8c12861d8: Network error: Client connection negotiation failed: client connection to 127.21.12.129:41895: connect: Connection refused (error 111)
I20260806 06:26:53.529626 21554 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHhLhT/build/release/bin/kudu with pid 31957
I20260806 06:26:53.542273 21554 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHhLhT/build/release/bin/kudu with pid 30962
I20260806 06:26:53.554275 21554 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHhLhT/build/release/bin/kudu with pid 31242
2026-08-06T06:26:53Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (6822 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785997581049705-21554-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 06:26:53 dist-test-slave-4tdj krb5kdc[32356](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 06:26:53 dist-test-slave-4tdj krb5kdc[32356](info): set up 2 sockets
Aug 06 06:26:53 dist-test-slave-4tdj krb5kdc[32356](info): commencing operation
krb5kdc: starting...
W20260806 06:26:54.671823 21554 mini_kdc.cc:121] Time spent starting KDC: real 1.094s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785997581049705-21554-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785997581049705-21554-0/krb5kdc/test-user.keytab.
Aug 06 06:26:54 dist-test-slave-4tdj krb5kdc[32356](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997614, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T06:26:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T06:26:54Z Disabled control of system clock
WARNING: no policy specified for oryx/127.21.12.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.12.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.12.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785997581049705-21554-0/krb5kdc/oryx_127.21.12.190.keytab.
Entry for principal oryx/127.21.12.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785997581049705-21554-0/krb5kdc/oryx_127.21.12.190.keytab.
I20260806 06:26:54.784904 21554 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHhLhT/build/release/bin/kudu
/tmp/dist-test-taskFHhLhT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785997581049705-21554-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785997581049705-21554-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785997581049705-21554-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785997581049705-21554-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.12.190:33805
--webserver_interface=127.21.12.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785997581049705-21554-0/krb5kdc/oryx_127.21.12.190.keytab
--principal=oryx/127.21.12.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.148:44455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.12.190:33805
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785997581049705-21554-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785997581049705-21554-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785997581049705-21554-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 06:26:54.863317 32378 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 06:26:54.863562 32378 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 06:26:54.863585 32378 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 06:26:54.864686 32378 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 06:26:54.864717 32378 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260806 06:26:54.864729 32378 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 06:26:54.864740 32378 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 06:26:54.864751 32378 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 06:26:54.864763 32378 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 06:26:54.866256 32378 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.148:44455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785997581049705-21554-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785997581049705-21554-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.12.190:33805
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785997581049705-21554-0/krb5kdc/oryx_127.21.12.190.keytab
--principal=oryx/127.21.12.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.12.190:33805
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785997581049705-21554-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.12.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32378
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785997581049705-21554-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 05:11:02 UTC on ccycloud.quasar-hvzbbz.root.comops.site
build id 4569617
I20260806 06:26:54.866497 32378 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 06:26:54.866667 32378 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 06:26:54.871296 32388 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:26:54.871309 32390 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:26:54.871296 32385 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:26:54.871551 32378 server_base.cc:1034] running on GCE node
I20260806 06:26:54.871835 32378 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 06:26:54.872097 32378 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 06:26:54.873242 32378 hybrid_clock.cc:648] HybridClock initialized: now 1785997614873230 us; error 32 us; skew 500 ppm
Aug 06 06:26:54 dist-test-slave-4tdj krb5kdc[32356](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997614, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.12.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 06:26:54.878819 32378 init.cc:377] Logged in from keytab as oryx/127.21.12.190@KRBTEST.COM (short username oryx)
I20260806 06:26:54.879269 32378 webserver.cc:460] Webserver started at http://127.21.12.190:44129/ using document root <none> and password file <none>
I20260806 06:26:54.879493 32378 fs_manager.cc:359] Metadata directory not provided
I20260806 06:26:54.879551 32378 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 06:26:54.879680 32378 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 06:26:54.880717 32378 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785997581049705-21554-0/minicluster-data/master-0/data/instance:
uuid: "987ca9a955c54e639f87808644890c4c"
format_stamp: "Formatted at 2026-08-06 06:26:54 on dist-test-slave-4tdj"
server_key: "e586081e60df8b4e136a182678b3ada3"
server_key_iv: "1fcbe59edc4066d09eda695d8051c96c"
server_key_version: "encryptionkey@0"
I20260806 06:26:54.881065 32378 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785997581049705-21554-0/minicluster-data/master-0/wal/instance:
uuid: "987ca9a955c54e639f87808644890c4c"
format_stamp: "Formatted at 2026-08-06 06:26:54 on dist-test-slave-4tdj"
server_key: "e586081e60df8b4e136a182678b3ada3"
server_key_iv: "1fcbe59edc4066d09eda695d8051c96c"
server_key_version: "encryptionkey@0"
I20260806 06:26:54.882364 32378 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260806 06:26:54.883105 32401 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 06:26:54.883270 32378 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260806 06:26:54.883356 32378 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785997581049705-21554-0/minicluster-data/master-0/data,/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785997581049705-21554-0/minicluster-data/master-0/wal
uuid: "987ca9a955c54e639f87808644890c4c"
format_stamp: "Formatted at 2026-08-06 06:26:54 on dist-test-slave-4tdj"
server_key: "e586081e60df8b4e136a182678b3ada3"
server_key_iv: "1fcbe59edc4066d09eda695d8051c96c"
server_key_version: "encryptionkey@0"
I20260806 06:26:54.883423 32378 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785997581049705-21554-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785997581049705-21554-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785997581049705-21554-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 06:26:54.893357 32378 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 06:26:54.895360 32378 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 06:26:54.895491 32378 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 06:26:54.899899 32378 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.190:33805
I20260806 06:26:54.899957 32508 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.190:33805 every 8 connection(s)
I20260806 06:26:54.900255 32378 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785997581049705-21554-0/minicluster-data/master-0/data/info.pb
I20260806 06:26:54.900880 32510 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 06:26:54.903127 32510 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 987ca9a955c54e639f87808644890c4c: Bootstrap starting.
I20260806 06:26:54.903692 32510 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 987ca9a955c54e639f87808644890c4c: Neither blocks nor log segments found. Creating new log.
I20260806 06:26:54.903945 32510 log.cc:824] T 00000000000000000000000000000000 P 987ca9a955c54e639f87808644890c4c: Log is configured to *not* fsync() on all Append() calls
I20260806 06:26:54.904592 32510 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 987ca9a955c54e639f87808644890c4c: No bootstrap required, opened a new log
I20260806 06:26:54.906627 32510 raft_consensus.cc:348] T 00000000000000000000000000000000 P 987ca9a955c54e639f87808644890c4c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "987ca9a955c54e639f87808644890c4c" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 33805 } }
I20260806 06:26:54.906760 32510 raft_consensus.cc:374] T 00000000000000000000000000000000 P 987ca9a955c54e639f87808644890c4c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 06:26:54.906791 32510 raft_consensus.cc:729] T 00000000000000000000000000000000 P 987ca9a955c54e639f87808644890c4c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 987ca9a955c54e639f87808644890c4c, State: Initialized, Role: FOLLOWER
I20260806 06:26:54.906912 32510 consensus_queue.cc:260] T 00000000000000000000000000000000 P 987ca9a955c54e639f87808644890c4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "987ca9a955c54e639f87808644890c4c" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 33805 } }
I20260806 06:26:54.906971 32510 raft_consensus.cc:388] T 00000000000000000000000000000000 P 987ca9a955c54e639f87808644890c4c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 06:26:54.907001 32510 raft_consensus.cc:482] T 00000000000000000000000000000000 P 987ca9a955c54e639f87808644890c4c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 06:26:54.907047 32510 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 987ca9a955c54e639f87808644890c4c [term 0 FOLLOWER]: Advancing to term 1
I20260806 06:26:54.907533 32510 raft_consensus.cc:504] T 00000000000000000000000000000000 P 987ca9a955c54e639f87808644890c4c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "987ca9a955c54e639f87808644890c4c" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 33805 } }
I20260806 06:26:54.907635 32510 leader_election.cc:302] T 00000000000000000000000000000000 P 987ca9a955c54e639f87808644890c4c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 987ca9a955c54e639f87808644890c4c; no voters:
I20260806 06:26:54.907784 32510 leader_election.cc:288] T 00000000000000000000000000000000 P 987ca9a955c54e639f87808644890c4c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 06:26:54.907830 32517 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 987ca9a955c54e639f87808644890c4c [term 1 FOLLOWER]: Leader election won for term 1
I20260806 06:26:54.907989 32517 raft_consensus.cc:686] T 00000000000000000000000000000000 P 987ca9a955c54e639f87808644890c4c [term 1 LEADER]: Becoming Leader. State: Replica: 987ca9a955c54e639f87808644890c4c, State: Running, Role: LEADER
I20260806 06:26:54.908030 32510 sys_catalog.cc:564] T 00000000000000000000000000000000 P 987ca9a955c54e639f87808644890c4c [sys.catalog]: configured and running, proceeding with master startup.
I20260806 06:26:54.908080 32517 consensus_queue.cc:237] T 00000000000000000000000000000000 P 987ca9a955c54e639f87808644890c4c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "987ca9a955c54e639f87808644890c4c" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 33805 } }
I20260806 06:26:54.908448 32520 sys_catalog.cc:455] T 00000000000000000000000000000000 P 987ca9a955c54e639f87808644890c4c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 987ca9a955c54e639f87808644890c4c. Latest consensus state: current_term: 1 leader_uuid: "987ca9a955c54e639f87808644890c4c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "987ca9a955c54e639f87808644890c4c" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 33805 } } }
I20260806 06:26:54.908445 32518 sys_catalog.cc:455] T 00000000000000000000000000000000 P 987ca9a955c54e639f87808644890c4c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "987ca9a955c54e639f87808644890c4c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "987ca9a955c54e639f87808644890c4c" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 33805 } } }
I20260806 06:26:54.908571 32520 sys_catalog.cc:458] T 00000000000000000000000000000000 P 987ca9a955c54e639f87808644890c4c [sys.catalog]: This master's current role is: LEADER
I20260806 06:26:54.908586 32518 sys_catalog.cc:458] T 00000000000000000000000000000000 P 987ca9a955c54e639f87808644890c4c [sys.catalog]: This master's current role is: LEADER
I20260806 06:26:54.908769 32533 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 06:26:54.909204 21554 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHhLhT/build/release/bin/kudu as pid 32378
I20260806 06:26:54.909284 32533 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 06:26:54.909315 21554 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785997581049705-21554-0/minicluster-data/master-0/wal/instance
I20260806 06:26:54.909533 21554 external_mini_cluster.cc:1428] Setting key cfac22344af5a1643940320c52998789
I20260806 06:26:54.910846 32533 catalog_manager.cc:1347] Generated new cluster ID: 1193d6cd30c2404584f333ac5f3bb6fc
I20260806 06:26:54.910900 32533 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 06 06:26:54 dist-test-slave-4tdj krb5kdc[32356](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997614, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.21.12.190@KRBTEST.COM
I20260806 06:26:54.920855 32549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:54.909995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33128 (local address 127.21.12.190:33805)
0806 06:26:54.910151 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0806 06:26:54.910161 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 06:26:54.910414 (+ 253us) server_negotiation.cc:407] Connection header received
0806 06:26:54.910549 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:26:54.910556 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:26:54.910614 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:26:54.911485 (+ 871us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:26:54.911587 (+ 102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:54.912928 (+ 1341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:54.913939 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:54.914570 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:54.914938 (+ 368us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:54.917244 (+ 2306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:26:54.917248 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:26:54.917249 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 06:26:54.917260 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:26:54.919069 (+ 1809us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:26:54.919418 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:26:54.919421 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:26:54.919421 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:26:54.919511 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:26:54.919690 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:26:54.919691 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:26:54.919691 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:26:54.919960 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:26:54.920432 (+ 472us) server_negotiation.cc:1035] Waiting for connection context
0806 06:26:54.920494 (+ 62us) server_negotiation.cc:299] Negotiation successful
0806 06:26:54.920633 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":83,"threads_started":1}
I20260806 06:26:54.924490 32533 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 06:26:54.924975 32533 catalog_manager.cc:1504] Loading token signing keys...
I20260806 06:26:54.929133 32533 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 987ca9a955c54e639f87808644890c4c: Generated new TSK 0
I20260806 06:26:54.929268 32533 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.21.12.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.12.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.12.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785997581049705-21554-0/krb5kdc/oryx_127.21.12.129.keytab.
Entry for principal oryx/127.21.12.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785997581049705-21554-0/krb5kdc/oryx_127.21.12.129.keytab.
I20260806 06:26:54.957669 21554 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHhLhT/build/release/bin/kudu
/tmp/dist-test-taskFHhLhT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785997581049705-21554-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785997581049705-21554-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785997581049705-21554-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785997581049705-21554-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.12.129:0
--local_ip_for_outbound_sockets=127.21.12.129
--webserver_interface=127.21.12.129
--webserver_port=0
--tserver_master_addrs=127.21.12.190:33805
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785997581049705-21554-0/krb5kdc/oryx_127.21.12.129.keytab
--principal=oryx/127.21.12.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.148:44455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785997581049705-21554-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785997581049705-21554-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785997581049705-21554-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 06:26:55.074383 32555 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 06:26:55.074653 32555 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 06:26:55.074692 32555 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 06:26:55.076134 32555 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 06:26:55.076185 32555 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 06:26:55.076207 32555 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 06:26:55.076225 32555 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.12.129
I20260806 06:26:55.077672 32555 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.148:44455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785997581049705-21554-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785997581049705-21554-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785997581049705-21554-0/krb5kdc/oryx_127.21.12.129.keytab
--principal=oryx/127.21.12.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.12.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785997581049705-21554-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.12.129
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.21.12.190:33805
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32555
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.12.129
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785997581049705-21554-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 05:11:02 UTC on ccycloud.quasar-hvzbbz.root.comops.site
build id 4569617
I20260806 06:26:55.077849 32555 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 06:26:55.078037 32555 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 06:26:55.082726 32566 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:26:55.082898 32555 server_base.cc:1034] running on GCE node
W20260806 06:26:55.082943 32567 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:26:55.082806 32569 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:26:55.083222 32555 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 06:26:55.083439 32555 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 06:26:55.084589 32555 hybrid_clock.cc:648] HybridClock initialized: now 1785997615084565 us; error 46 us; skew 500 ppm
Aug 06 06:26:55 dist-test-slave-4tdj krb5kdc[32356](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997615, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.12.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 06:26:55.087132 32555 init.cc:377] Logged in from keytab as oryx/127.21.12.129@KRBTEST.COM (short username oryx)
I20260806 06:26:55.087548 32555 webserver.cc:460] Webserver started at http://127.21.12.129:35617/ using document root <none> and password file <none>
I20260806 06:26:55.087738 32555 fs_manager.cc:359] Metadata directory not provided
I20260806 06:26:55.087785 32555 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 06:26:55.087906 32555 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 06:26:55.088861 32555 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785997581049705-21554-0/minicluster-data/ts-0/data/instance:
uuid: "067b2ec58c34466f99bd66c5780d8fd3"
format_stamp: "Formatted at 2026-08-06 06:26:55 on dist-test-slave-4tdj"
server_key: "8a500685f6a5360c5e1712837211b4bc"
server_key_iv: "4e76ce96afb64a7b982edbf9116145b2"
server_key_version: "encryptionkey@0"
I20260806 06:26:55.089183 32555 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785997581049705-21554-0/minicluster-data/ts-0/wal/instance:
uuid: "067b2ec58c34466f99bd66c5780d8fd3"
format_stamp: "Formatted at 2026-08-06 06:26:55 on dist-test-slave-4tdj"
server_key: "8a500685f6a5360c5e1712837211b4bc"
server_key_iv: "4e76ce96afb64a7b982edbf9116145b2"
server_key_version: "encryptionkey@0"
I20260806 06:26:55.090442 32555 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260806 06:26:55.091137 32583 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 06:26:55.091288 32555 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260806 06:26:55.091370 32555 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785997581049705-21554-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785997581049705-21554-0/minicluster-data/ts-0/wal
uuid: "067b2ec58c34466f99bd66c5780d8fd3"
format_stamp: "Formatted at 2026-08-06 06:26:55 on dist-test-slave-4tdj"
server_key: "8a500685f6a5360c5e1712837211b4bc"
server_key_iv: "4e76ce96afb64a7b982edbf9116145b2"
server_key_version: "encryptionkey@0"
I20260806 06:26:55.091434 32555 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785997581049705-21554-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785997581049705-21554-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785997581049705-21554-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 06:26:55.109097 32555 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 06:26:55.111029 32555 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 06:26:55.111156 32555 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 06:26:55.111606 32555 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 06:26:55.111649 32555 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 06:26:55.111677 32555 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 06:26:55.111694 32555 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 06:26:55 dist-test-slave-4tdj krb5kdc[32356](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997615, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.12.129@KRBTEST.COM for oryx/127.21.12.190@KRBTEST.COM
I20260806 06:26:55.118145 32555 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.129:44983
I20260806 06:26:55.118224 32760 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.129:44983 every 8 connection(s)
I20260806 06:26:55.118551 32555 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785997581049705-21554-0/minicluster-data/ts-0/data/info.pb
I20260806 06:26:55.120364 32549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:55.112043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.129:54269 (local address 127.21.12.190:33805)
0806 06:26:55.112085 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0806 06:26:55.112088 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 06:26:55.112299 (+ 211us) server_negotiation.cc:407] Connection header received
0806 06:26:55.112812 (+ 513us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:26:55.112813 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:26:55.112832 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:26:55.112890 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:26:55.113744 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:55.114536 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:55.115485 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:55.115705 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:55.115759 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:55.117846 (+ 2087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:26:55.117847 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:26:55.117848 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 06:26:55.117855 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:26:55.119331 (+ 1476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:26:55.119687 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:26:55.119688 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:26:55.119689 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:26:55.119746 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:26:55.119920 (+ 174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:26:55.119920 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:26:55.119920 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:26:55.120012 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:26:55.120055 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0806 06:26:55.120237 (+ 182us) server_negotiation.cc:299] Negotiation successful
0806 06:26:55.120302 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260806 06:26:55.120520 32595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:55.112044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:33805 (local address 127.21.12.129:54269)
0806 06:26:55.112191 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0806 06:26:55.112225 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0806 06:26:55.112714 (+ 489us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:26:55.113008 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:26:55.113013 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:26:55.113357 (+ 344us) client_negotiation.cc:189] Negotiated authn=SASL
0806 06:26:55.113628 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:55.113634 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:55.114618 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:55.114619 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:55.115401 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:55.115404 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:55.115785 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:55.115786 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:55.115908 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:55.116348 (+ 440us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 06:26:55.116357 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0806 06:26:55.117756 (+ 1399us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 06:26:55.119399 (+ 1643us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:26:55.119404 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:26:55.119406 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:26:55.119634 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:26:55.119796 (+ 162us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:26:55.119796 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:26:55.119797 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:26:55.119885 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:26:55.120051 (+ 166us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 06:26:55.120054 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 06:26:55.120153 (+ 99us) client_negotiation.cc:769] Sending connection context
0806 06:26:55.120225 (+ 72us) client_negotiation.cc:240] Negotiation successful
0806 06:26:55.120334 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":77,"threads_started":1}
I20260806 06:26:55.120995 32761 heartbeater.cc:347] Connected to a master server at 127.21.12.190:33805
I20260806 06:26:55.121080 32761 heartbeater.cc:464] Registering TS with master...
I20260806 06:26:55.121255 32761 heartbeater.cc:511] Master 127.21.12.190:33805 requested a full tablet report, sending...
I20260806 06:26:55.121759 32424 ts_manager.cc:194] Registered new tserver with Master: 067b2ec58c34466f99bd66c5780d8fd3 (127.21.12.129:44983)
I20260806 06:26:55.122422 32424 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.21.12.129@KRBTEST.COM'} at 127.21.12.129:54269
I20260806 06:26:55.123467 21554 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHhLhT/build/release/bin/kudu as pid 32555
I20260806 06:26:55.123566 21554 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785997581049705-21554-0/minicluster-data/ts-0/wal/instance
I20260806 06:26:55.123750 21554 external_mini_cluster.cc:1428] Setting key a07a2cafdc8f1c26743d38a9583b9e96
I20260806 06:26:55.127782 301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:55.122462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:33805 (local address 127.21.12.129:56285)
0806 06:26:55.122632 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0806 06:26:55.122642 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0806 06:26:55.122698 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:26:55.122821 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:26:55.122821 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:26:55.123031 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0806 06:26:55.123099 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:55.123102 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:55.123775 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:55.123776 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:55.124402 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:55.124405 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:55.124709 (+ 304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:55.124710 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:55.124767 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:55.125104 (+ 337us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 06:26:55.125109 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0806 06:26:55.125435 (+ 326us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 06:26:55.126972 (+ 1537us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:26:55.126973 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:26:55.126973 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:26:55.127122 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:26:55.127299 (+ 177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:26:55.127300 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:26:55.127300 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:26:55.127343 (+ 43us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:26:55.127567 (+ 224us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 06:26:55.127568 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 06:26:55.127602 (+ 34us) client_negotiation.cc:769] Sending connection context
0806 06:26:55.127649 (+ 47us) client_negotiation.cc:240] Negotiation successful
0806 06:26:55.127715 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":115,"threads_started":1}
I20260806 06:26:55.127826 32549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:55.122535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.129:56285 (local address 127.21.12.190:33805)
0806 06:26:55.122575 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0806 06:26:55.122577 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:26:55.122704 (+ 127us) server_negotiation.cc:407] Connection header received
0806 06:26:55.122749 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:26:55.122750 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:26:55.122762 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:26:55.122814 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:26:55.123175 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:55.123687 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:55.124470 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:55.124647 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:55.124695 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:55.125523 (+ 828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:26:55.125524 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:26:55.125524 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 06:26:55.125530 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:26:55.126899 (+ 1369us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:26:55.127186 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:26:55.127187 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:26:55.127187 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:26:55.127244 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:26:55.127392 (+ 148us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:26:55.127393 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:26:55.127393 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:26:55.127514 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:26:55.127574 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0806 06:26:55.127673 (+ 99us) server_negotiation.cc:299] Negotiation successful
0806 06:26:55.127751 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
WARNING: no policy specified for oryx/127.21.12.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.12.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.12.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785997581049705-21554-0/krb5kdc/oryx_127.21.12.130.keytab.
Entry for principal oryx/127.21.12.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785997581049705-21554-0/krb5kdc/oryx_127.21.12.130.keytab.
I20260806 06:26:55.149094 21554 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHhLhT/build/release/bin/kudu
/tmp/dist-test-taskFHhLhT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785997581049705-21554-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785997581049705-21554-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785997581049705-21554-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785997581049705-21554-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.12.130:0
--local_ip_for_outbound_sockets=127.21.12.130
--webserver_interface=127.21.12.130
--webserver_port=0
--tserver_master_addrs=127.21.12.190:33805
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785997581049705-21554-0/krb5kdc/oryx_127.21.12.130.keytab
--principal=oryx/127.21.12.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.148:44455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785997581049705-21554-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785997581049705-21554-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785997581049705-21554-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 06:26:55.228411 304 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 06:26:55.228608 304 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 06:26:55.228631 304 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 06:26:55.229714 304 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 06:26:55.229746 304 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 06:26:55.229765 304 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 06:26:55.229781 304 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.12.130
I20260806 06:26:55.231243 304 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.148:44455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785997581049705-21554-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785997581049705-21554-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785997581049705-21554-0/krb5kdc/oryx_127.21.12.130.keytab
--principal=oryx/127.21.12.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.12.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785997581049705-21554-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.12.130
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.21.12.190:33805
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.304
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.12.130
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785997581049705-21554-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 05:11:02 UTC on ccycloud.quasar-hvzbbz.root.comops.site
build id 4569617
I20260806 06:26:55.231417 304 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 06:26:55.231596 304 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 06:26:55.236282 310 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:26:55.236403 304 server_base.cc:1034] running on GCE node
W20260806 06:26:55.236398 309 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:26:55.236455 312 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:26:55.236760 304 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 06:26:55.236992 304 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 06:26:55.238159 304 hybrid_clock.cc:648] HybridClock initialized: now 1785997615238144 us; error 28 us; skew 500 ppm
Aug 06 06:26:55 dist-test-slave-4tdj krb5kdc[32356](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997615, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.12.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 06:26:55.240556 304 init.cc:377] Logged in from keytab as oryx/127.21.12.130@KRBTEST.COM (short username oryx)
I20260806 06:26:55.240958 304 webserver.cc:460] Webserver started at http://127.21.12.130:33869/ using document root <none> and password file <none>
I20260806 06:26:55.241175 304 fs_manager.cc:359] Metadata directory not provided
I20260806 06:26:55.241222 304 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 06:26:55.241343 304 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 06:26:55.242333 304 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785997581049705-21554-0/minicluster-data/ts-1/data/instance:
uuid: "52f06bd6d6a448aeb07ada8c0d89e2e1"
format_stamp: "Formatted at 2026-08-06 06:26:55 on dist-test-slave-4tdj"
server_key: "7095d4b1a8fd8cde1357c1d3ad6ed681"
server_key_iv: "5df8e5fb0e73ea97464ca49ac8937b83"
server_key_version: "encryptionkey@0"
I20260806 06:26:55.242657 304 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785997581049705-21554-0/minicluster-data/ts-1/wal/instance:
uuid: "52f06bd6d6a448aeb07ada8c0d89e2e1"
format_stamp: "Formatted at 2026-08-06 06:26:55 on dist-test-slave-4tdj"
server_key: "7095d4b1a8fd8cde1357c1d3ad6ed681"
server_key_iv: "5df8e5fb0e73ea97464ca49ac8937b83"
server_key_version: "encryptionkey@0"
I20260806 06:26:55.243836 304 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260806 06:26:55.244555 328 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 06:26:55.244700 304 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260806 06:26:55.244779 304 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785997581049705-21554-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785997581049705-21554-0/minicluster-data/ts-1/wal
uuid: "52f06bd6d6a448aeb07ada8c0d89e2e1"
format_stamp: "Formatted at 2026-08-06 06:26:55 on dist-test-slave-4tdj"
server_key: "7095d4b1a8fd8cde1357c1d3ad6ed681"
server_key_iv: "5df8e5fb0e73ea97464ca49ac8937b83"
server_key_version: "encryptionkey@0"
I20260806 06:26:55.244843 304 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785997581049705-21554-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785997581049705-21554-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785997581049705-21554-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 06:26:55.253830 304 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 06:26:55.255537 304 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 06:26:55.255661 304 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 06:26:55.256148 304 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 06:26:55.256187 304 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 06:26:55.256213 304 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 06:26:55.256238 304 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 06:26:55 dist-test-slave-4tdj krb5kdc[32356](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997615, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.12.130@KRBTEST.COM for oryx/127.21.12.190@KRBTEST.COM
I20260806 06:26:55.262671 304 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.130:39591
I20260806 06:26:55.262840 505 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.130:39591 every 8 connection(s)
I20260806 06:26:55.263008 304 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785997581049705-21554-0/minicluster-data/ts-1/data/info.pb
I20260806 06:26:55.263502 21554 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHhLhT/build/release/bin/kudu as pid 304
I20260806 06:26:55.263610 21554 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785997581049705-21554-0/minicluster-data/ts-1/wal/instance
I20260806 06:26:55.263792 21554 external_mini_cluster.cc:1428] Setting key 5abffe9b82d7a6f4397debf98744fcab
I20260806 06:26:55.265050 32549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:55.256477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.130:51353 (local address 127.21.12.190:33805)
0806 06:26:55.256540 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0806 06:26:55.256543 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 06:26:55.256732 (+ 189us) server_negotiation.cc:407] Connection header received
0806 06:26:55.257189 (+ 457us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:26:55.257190 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:26:55.257204 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:26:55.257254 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:26:55.257967 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:55.258732 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:55.259767 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:55.259950 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:55.260012 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:55.262346 (+ 2334us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:26:55.262348 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:26:55.262348 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 06:26:55.262355 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:26:55.263800 (+ 1445us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:26:55.264186 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:26:55.264188 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:26:55.264188 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:26:55.264249 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:26:55.264459 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:26:55.264460 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:26:55.264460 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:26:55.264570 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:26:55.264625 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0806 06:26:55.264874 (+ 249us) server_negotiation.cc:299] Negotiation successful
0806 06:26:55.264953 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260806 06:26:55.265178 338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:55.256446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:33805 (local address 127.21.12.130:51353)
0806 06:26:55.256615 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0806 06:26:55.256653 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0806 06:26:55.257096 (+ 443us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:26:55.257315 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:26:55.257318 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:26:55.257606 (+ 288us) client_negotiation.cc:189] Negotiated authn=SASL
0806 06:26:55.257862 (+ 256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:55.257868 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:55.258851 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:55.258852 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:55.259681 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:55.259684 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:55.260032 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:55.260033 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:55.260150 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:55.260566 (+ 416us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 06:26:55.260574 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0806 06:26:55.262244 (+ 1670us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 06:26:55.263867 (+ 1623us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:26:55.263875 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:26:55.263877 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:26:55.264117 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:26:55.264302 (+ 185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:26:55.264303 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:26:55.264303 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:26:55.264405 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:26:55.264626 (+ 221us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 06:26:55.264633 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 06:26:55.264758 (+ 125us) client_negotiation.cc:769] Sending connection context
0806 06:26:55.264841 (+ 83us) client_negotiation.cc:240] Negotiation successful
0806 06:26:55.264984 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":105,"threads_started":1}
I20260806 06:26:55.266378 506 heartbeater.cc:347] Connected to a master server at 127.21.12.190:33805
I20260806 06:26:55.266467 506 heartbeater.cc:464] Registering TS with master...
I20260806 06:26:55.266682 506 heartbeater.cc:511] Master 127.21.12.190:33805 requested a full tablet report, sending...
I20260806 06:26:55.267364 32425 ts_manager.cc:194] Registered new tserver with Master: 52f06bd6d6a448aeb07ada8c0d89e2e1 (127.21.12.130:39591)
I20260806 06:26:55.267853 32425 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.21.12.130@KRBTEST.COM'} at 127.21.12.130:51353
I20260806 06:26:55.272869 514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:55.267279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:33805 (local address 127.21.12.130:54469)
0806 06:26:55.267429 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0806 06:26:55.267439 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0806 06:26:55.267498 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:26:55.267656 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:26:55.267657 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:26:55.267839 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0806 06:26:55.267893 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:55.267895 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:55.268713 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:55.268714 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:55.269281 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:55.269283 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:55.269741 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:55.269742 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:55.269794 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:55.270148 (+ 354us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 06:26:55.270152 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0806 06:26:55.270443 (+ 291us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 06:26:55.271931 (+ 1488us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:26:55.271932 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:26:55.271932 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:26:55.272106 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:26:55.272308 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:26:55.272309 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:26:55.272309 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:26:55.272355 (+ 46us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:26:55.272658 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 06:26:55.272658 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 06:26:55.272692 (+ 34us) client_negotiation.cc:769] Sending connection context
0806 06:26:55.272738 (+ 46us) client_negotiation.cc:240] Negotiation successful
0806 06:26:55.272808 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":59,"threads_started":1}
WARNING: no policy specified for oryx/127.21.12.131@KRBTEST.COM; defaulting to no policy
I20260806 06:26:55.272915 32549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:55.267329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.130:54469 (local address 127.21.12.190:33805)
0806 06:26:55.267367 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0806 06:26:55.267369 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:26:55.267492 (+ 123us) server_negotiation.cc:407] Connection header received
0806 06:26:55.267544 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:26:55.267545 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:26:55.267556 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:26:55.267646 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:26:55.267996 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:55.268600 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:55.269372 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:55.269668 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:55.269742 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:55.270516 (+ 774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:26:55.270518 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:26:55.270518 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 06:26:55.270524 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:26:55.271842 (+ 1318us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:26:55.272177 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:26:55.272178 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:26:55.272179 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:26:55.272245 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:26:55.272406 (+ 161us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:26:55.272407 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:26:55.272407 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:26:55.272598 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:26:55.272669 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0806 06:26:55.272762 (+ 93us) server_negotiation.cc:299] Negotiation successful
0806 06:26:55.272832 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.12.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.12.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785997581049705-21554-0/krb5kdc/oryx_127.21.12.131.keytab.
Entry for principal oryx/127.21.12.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785997581049705-21554-0/krb5kdc/oryx_127.21.12.131.keytab.
I20260806 06:26:55.288648 21554 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHhLhT/build/release/bin/kudu
/tmp/dist-test-taskFHhLhT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785997581049705-21554-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785997581049705-21554-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785997581049705-21554-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785997581049705-21554-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.12.131:0
--local_ip_for_outbound_sockets=127.21.12.131
--webserver_interface=127.21.12.131
--webserver_port=0
--tserver_master_addrs=127.21.12.190:33805
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785997581049705-21554-0/krb5kdc/oryx_127.21.12.131.keytab
--principal=oryx/127.21.12.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.148:44455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785997581049705-21554-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785997581049705-21554-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785997581049705-21554-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 06:26:55.367642 518 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 06:26:55.367826 518 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 06:26:55.367848 518 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 06:26:55.368923 518 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 06:26:55.368955 518 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 06:26:55.368974 518 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 06:26:55.368990 518 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.12.131
I20260806 06:26:55.370532 518 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.148:44455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785997581049705-21554-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785997581049705-21554-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785997581049705-21554-0/krb5kdc/oryx_127.21.12.131.keytab
--principal=oryx/127.21.12.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.12.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785997581049705-21554-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.12.131
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.21.12.190:33805
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.518
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.12.131
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785997581049705-21554-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 05:11:02 UTC on ccycloud.quasar-hvzbbz.root.comops.site
build id 4569617
I20260806 06:26:55.370715 518 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 06:26:55.370874 518 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260806 06:26:55.375540 518 server_base.cc:1034] running on GCE node
W20260806 06:26:55.375504 528 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:26:55.375589 526 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:26:55.375501 533 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:26:55.375905 518 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 06:26:55.376150 518 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 06:26:55.377286 518 hybrid_clock.cc:648] HybridClock initialized: now 1785997615377276 us; error 29 us; skew 500 ppm
Aug 06 06:26:55 dist-test-slave-4tdj krb5kdc[32356](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997615, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.12.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 06:26:55.379652 518 init.cc:377] Logged in from keytab as oryx/127.21.12.131@KRBTEST.COM (short username oryx)
I20260806 06:26:55.380069 518 webserver.cc:460] Webserver started at http://127.21.12.131:45575/ using document root <none> and password file <none>
I20260806 06:26:55.380260 518 fs_manager.cc:359] Metadata directory not provided
I20260806 06:26:55.380306 518 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 06:26:55.380427 518 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 06:26:55.381397 518 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785997581049705-21554-0/minicluster-data/ts-2/data/instance:
uuid: "8483d5dffddc4ca080e13001a1707d12"
format_stamp: "Formatted at 2026-08-06 06:26:55 on dist-test-slave-4tdj"
server_key: "b116fc0c5cb4f3afa7465a461abc3c5c"
server_key_iv: "13c9b05357d1ece9ad307fb3e9584360"
server_key_version: "encryptionkey@0"
I20260806 06:26:55.381770 518 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785997581049705-21554-0/minicluster-data/ts-2/wal/instance:
uuid: "8483d5dffddc4ca080e13001a1707d12"
format_stamp: "Formatted at 2026-08-06 06:26:55 on dist-test-slave-4tdj"
server_key: "b116fc0c5cb4f3afa7465a461abc3c5c"
server_key_iv: "13c9b05357d1ece9ad307fb3e9584360"
server_key_version: "encryptionkey@0"
I20260806 06:26:55.383047 518 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260806 06:26:55.383677 545 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 06:26:55.383831 518 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260806 06:26:55.383916 518 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785997581049705-21554-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785997581049705-21554-0/minicluster-data/ts-2/wal
uuid: "8483d5dffddc4ca080e13001a1707d12"
format_stamp: "Formatted at 2026-08-06 06:26:55 on dist-test-slave-4tdj"
server_key: "b116fc0c5cb4f3afa7465a461abc3c5c"
server_key_iv: "13c9b05357d1ece9ad307fb3e9584360"
server_key_version: "encryptionkey@0"
I20260806 06:26:55.383970 518 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785997581049705-21554-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785997581049705-21554-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785997581049705-21554-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 06:26:55.411559 518 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 06:26:55.413645 518 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 06:26:55.413779 518 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 06:26:55.414314 518 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 06:26:55.414367 518 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 06:26:55.414397 518 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 06:26:55.414422 518 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 06:26:55 dist-test-slave-4tdj krb5kdc[32356](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997615, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.12.131@KRBTEST.COM for oryx/127.21.12.190@KRBTEST.COM
I20260806 06:26:55.421324 518 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.131:41903
I20260806 06:26:55.421386 714 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.131:41903 every 8 connection(s)
I20260806 06:26:55.421729 518 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785997581049705-21554-0/minicluster-data/ts-2/data/info.pb
I20260806 06:26:55.423238 21554 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHhLhT/build/release/bin/kudu as pid 518
I20260806 06:26:55.423331 21554 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785997581049705-21554-0/minicluster-data/ts-2/wal/instance
I20260806 06:26:55.423497 21554 external_mini_cluster.cc:1428] Setting key 9b3cd626769ed9858d6c706c30961676
I20260806 06:26:55.424374 32549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:55.414769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.131:52777 (local address 127.21.12.190:33805)
0806 06:26:55.414833 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0806 06:26:55.414836 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 06:26:55.415040 (+ 204us) server_negotiation.cc:407] Connection header received
0806 06:26:55.415720 (+ 680us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:26:55.415722 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:26:55.415735 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:26:55.415801 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:26:55.416624 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:55.417387 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:55.418870 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:55.419085 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:55.419145 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:55.421509 (+ 2364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:26:55.421511 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:26:55.421511 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 06:26:55.421519 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:26:55.422950 (+ 1431us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:26:55.423357 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:26:55.423358 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:26:55.423359 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:26:55.423426 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:26:55.423683 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:26:55.423684 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:26:55.423684 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:26:55.423835 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:26:55.423900 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0806 06:26:55.424198 (+ 298us) server_negotiation.cc:299] Negotiation successful
0806 06:26:55.424281 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260806 06:26:55.424525 560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:55.414747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:33805 (local address 127.21.12.131:52777)
0806 06:26:55.414902 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0806 06:26:55.414951 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0806 06:26:55.415610 (+ 659us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:26:55.415880 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:26:55.415884 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:26:55.416228 (+ 344us) client_negotiation.cc:189] Negotiated authn=SASL
0806 06:26:55.416526 (+ 298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:55.416534 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:55.417484 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:55.417486 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:55.418785 (+ 1299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:55.418788 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:55.419188 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:55.419189 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:55.419298 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:55.419740 (+ 442us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 06:26:55.419757 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 06:26:55.421393 (+ 1636us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 06:26:55.423038 (+ 1645us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:26:55.423045 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:26:55.423046 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:26:55.423259 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:26:55.423484 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:26:55.423484 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:26:55.423485 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:26:55.423622 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:26:55.423898 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 06:26:55.423904 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 06:26:55.424062 (+ 158us) client_negotiation.cc:769] Sending connection context
0806 06:26:55.424171 (+ 109us) client_negotiation.cc:240] Negotiation successful
0806 06:26:55.424335 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":76,"threads_started":1}
I20260806 06:26:55.424935 715 heartbeater.cc:347] Connected to a master server at 127.21.12.190:33805
I20260806 06:26:55.425016 715 heartbeater.cc:464] Registering TS with master...
I20260806 06:26:55.425195 715 heartbeater.cc:511] Master 127.21.12.190:33805 requested a full tablet report, sending...
I20260806 06:26:55.425602 32424 ts_manager.cc:194] Registered new tserver with Master: 8483d5dffddc4ca080e13001a1707d12 (127.21.12.131:41903)
I20260806 06:26:55.426076 32424 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.21.12.131@KRBTEST.COM'} at 127.21.12.131:52777
I20260806 06:26:55.430747 32549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:55.426020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.131:41411 (local address 127.21.12.190:33805)
0806 06:26:55.426078 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0806 06:26:55.426080 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:26:55.426130 (+ 50us) server_negotiation.cc:407] Connection header received
0806 06:26:55.426216 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:26:55.426217 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:26:55.426227 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:26:55.426271 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:26:55.426641 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:55.427079 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:55.427695 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:55.427866 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:55.427906 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:55.428637 (+ 731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:26:55.428638 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:26:55.428638 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 06:26:55.428642 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:26:55.429812 (+ 1170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:26:55.430107 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:26:55.430108 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:26:55.430108 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:26:55.430155 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:26:55.430326 (+ 171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:26:55.430327 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:26:55.430328 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:26:55.430450 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:26:55.430520 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0806 06:26:55.430628 (+ 108us) server_negotiation.cc:299] Negotiation successful
0806 06:26:55.430689 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260806 06:26:55.430751 723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:55.425954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:33805 (local address 127.21.12.131:41411)
0806 06:26:55.426080 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0806 06:26:55.426090 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0806 06:26:55.426143 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:26:55.426274 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:26:55.426275 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:26:55.426482 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0806 06:26:55.426553 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:55.426555 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:55.427157 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:55.427158 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:55.427639 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:55.427641 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:55.427911 (+ 270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:55.427911 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:55.427956 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:55.428312 (+ 356us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 06:26:55.428317 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0806 06:26:55.428577 (+ 260us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 06:26:55.429892 (+ 1315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:26:55.429892 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:26:55.429893 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:26:55.430044 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:26:55.430191 (+ 147us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:26:55.430191 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:26:55.430191 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:26:55.430233 (+ 42us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:26:55.430514 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 06:26:55.430515 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 06:26:55.430549 (+ 34us) client_negotiation.cc:769] Sending connection context
0806 06:26:55.430604 (+ 55us) client_negotiation.cc:240] Negotiation successful
0806 06:26:55.430683 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":72,"threads_started":1}
I20260806 06:26:55.434826 21554 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 06 06:26:55 dist-test-slave-4tdj krb5kdc[32356](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997615, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 06 06:26:55 dist-test-slave-4tdj krb5kdc[32356](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.21.12.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 06:26:55 dist-test-slave-4tdj krb5kdc[32356](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.21.12.190@KRBTEST.COM, Server not found in Kerberos database
W20260806 06:26:55.455854 32549 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 06:26:55.450952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33136 (local address 127.21.12.190:33805)
0806 06:26:55.451009 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0806 06:26:55.451011 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:26:55.451020 (+ 9us) server_negotiation.cc:407] Connection header received
0806 06:26:55.451045 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:26:55.451046 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:26:55.451055 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:26:55.451105 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:26:55.451615 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:55.452091 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:55.452805 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:55.452996 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:55.453065 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:55.455786 (+ 2721us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33136: BlockingRecv error: recv got EOF from 127.0.0.1:33136 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
Aug 06 06:26:55 dist-test-slave-4tdj krb5kdc[32356](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997615, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.21.12.190@KRBTEST.COM
I20260806 06:26:55.462698 32549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:55.456554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33140 (local address 127.21.12.190:33805)
0806 06:26:55.456614 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0806 06:26:55.456616 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:26:55.456674 (+ 58us) server_negotiation.cc:407] Connection header received
0806 06:26:55.456729 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:26:55.456730 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:26:55.456740 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:26:55.456781 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:26:55.457209 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:55.457657 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:55.458279 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:55.458495 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:55.458547 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:55.460312 (+ 1765us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:26:55.460313 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:26:55.460314 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 06:26:55.460320 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:26:55.461463 (+ 1143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:26:55.461821 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:26:55.461823 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:26:55.461823 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:26:55.461907 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:26:55.462096 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:26:55.462097 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:26:55.462097 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:26:55.462246 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:26:55.462376 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0806 06:26:55.462534 (+ 158us) server_negotiation.cc:299] Negotiation successful
0806 06:26:55.462609 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260806 06:26:55.464263 32424 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33140:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 06:26:55.465807 32424 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 06:26:55.472088 753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:55.468088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39336 (local address 127.21.12.130:39591)
0806 06:26:55.468556 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0806 06:26:55.468560 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 06:26:55.468569 (+ 9us) server_negotiation.cc:407] Connection header received
0806 06:26:55.468603 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:26:55.468605 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:26:55.468620 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:26:55.468760 (+ 140us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 06:26:55.468925 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:55.469999 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:55.471138 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:55.471704 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:55.471770 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:55.471852 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0806 06:26:55.471952 (+ 100us) server_negotiation.cc:299] Negotiation successful
0806 06:26:55.471990 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":433,"thread_start_us":101,"threads_started":1}
I20260806 06:26:55.472077 756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:55.467976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.130:39591 (local address 127.0.0.1:39336)
0806 06:26:55.468311 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0806 06:26:55.468352 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0806 06:26:55.468418 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:26:55.468820 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:26:55.468821 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:26:55.468823 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 06:26:55.468857 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:55.468858 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:55.470101 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:55.470102 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:55.471066 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:55.471068 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:55.471784 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:55.471785 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:55.471861 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:55.471876 (+ 15us) client_negotiation.cc:769] Sending connection context
0806 06:26:55.471940 (+ 64us) client_negotiation.cc:240] Negotiation successful
0806 06:26:55.471988 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"mutex_wait_us":32,"thread_start_us":53,"threads_started":1}
I20260806 06:26:55.472254 757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:55.468399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44776 (local address 127.21.12.131:41903)
0806 06:26:55.468581 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0806 06:26:55.468585 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 06:26:55.468593 (+ 8us) server_negotiation.cc:407] Connection header received
0806 06:26:55.468625 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:26:55.468626 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:26:55.468638 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:26:55.468706 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 06:26:55.468932 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:55.469737 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:55.471068 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:55.471752 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:55.471820 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:55.471889 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0806 06:26:55.472119 (+ 230us) server_negotiation.cc:299] Negotiation successful
0806 06:26:55.472162 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":84,"threads_started":1}
I20260806 06:26:55.472275 752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:55.467985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.129:44983 (local address 127.0.0.1:55198)
0806 06:26:55.468347 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0806 06:26:55.468358 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0806 06:26:55.468448 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:26:55.468731 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:26:55.468735 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:26:55.468737 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 06:26:55.468820 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:55.468822 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:55.470143 (+ 1321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:55.470144 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:55.471233 (+ 1089us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:55.471235 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:55.472074 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:55.472075 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:55.472150 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:55.472154 (+ 4us) client_negotiation.cc:769] Sending connection context
0806 06:26:55.472183 (+ 29us) client_negotiation.cc:240] Negotiation successful
0806 06:26:55.472215 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":343,"mutex_wait_us":3,"thread_start_us":80,"threads_started":1}
I20260806 06:26:55.472457 749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:55.468188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.131:41903 (local address 127.0.0.1:44776)
0806 06:26:55.468384 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0806 06:26:55.468392 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0806 06:26:55.468431 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:26:55.468770 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:26:55.468770 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:26:55.468773 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 06:26:55.468852 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:55.468854 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:55.469807 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:55.469808 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:55.470986 (+ 1178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:55.470988 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:55.471924 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:55.471925 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:55.471984 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:55.471989 (+ 5us) client_negotiation.cc:769] Sending connection context
0806 06:26:55.472040 (+ 51us) client_negotiation.cc:240] Negotiation successful
0806 06:26:55.472077 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185,"mutex_wait_us":20,"thread_start_us":66,"threads_started":1}
I20260806 06:26:55.472970 755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:55.468113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55198 (local address 127.21.12.129:44983)
0806 06:26:55.468301 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0806 06:26:55.468306 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 06:26:55.468419 (+ 113us) server_negotiation.cc:407] Connection header received
0806 06:26:55.468562 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:26:55.468565 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:26:55.468584 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:26:55.468899 (+ 315us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 06:26:55.468960 (+ 61us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:55.470070 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:55.471304 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:55.471978 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:55.472731 (+ 753us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:55.472815 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0806 06:26:55.472839 (+ 24us) server_negotiation.cc:299] Negotiation successful
0806 06:26:55.472880 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":85,"threads_started":1}
I20260806 06:26:55.473042 404 tablet_service.cc:1467] Processing CreateTablet for tablet 40ad6b3dc98b4aa58a267c0e671af97e (DEFAULT_TABLE table=test-table [id=5fe9e3990d3e41d6ae29f70066ff9dea]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 06:26:55.473150 614 tablet_service.cc:1467] Processing CreateTablet for tablet 40ad6b3dc98b4aa58a267c0e671af97e (DEFAULT_TABLE table=test-table [id=5fe9e3990d3e41d6ae29f70066ff9dea]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 06:26:55.473438 404 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 40ad6b3dc98b4aa58a267c0e671af97e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 06:26:55.473502 614 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 40ad6b3dc98b4aa58a267c0e671af97e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 06:26:55.473549 32651 tablet_service.cc:1467] Processing CreateTablet for tablet 40ad6b3dc98b4aa58a267c0e671af97e (DEFAULT_TABLE table=test-table [id=5fe9e3990d3e41d6ae29f70066ff9dea]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 06:26:55.473862 32651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 40ad6b3dc98b4aa58a267c0e671af97e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 06:26:55.476465 759 tablet_bootstrap.cc:492] T 40ad6b3dc98b4aa58a267c0e671af97e P 8483d5dffddc4ca080e13001a1707d12: Bootstrap starting.
I20260806 06:26:55.477046 759 tablet_bootstrap.cc:654] T 40ad6b3dc98b4aa58a267c0e671af97e P 8483d5dffddc4ca080e13001a1707d12: Neither blocks nor log segments found. Creating new log.
I20260806 06:26:55.477357 759 log.cc:824] T 40ad6b3dc98b4aa58a267c0e671af97e P 8483d5dffddc4ca080e13001a1707d12: Log is configured to *not* fsync() on all Append() calls
I20260806 06:26:55.477504 762 tablet_bootstrap.cc:492] T 40ad6b3dc98b4aa58a267c0e671af97e P 067b2ec58c34466f99bd66c5780d8fd3: Bootstrap starting.
I20260806 06:26:55.477576 761 tablet_bootstrap.cc:492] T 40ad6b3dc98b4aa58a267c0e671af97e P 52f06bd6d6a448aeb07ada8c0d89e2e1: Bootstrap starting.
I20260806 06:26:55.478047 762 tablet_bootstrap.cc:654] T 40ad6b3dc98b4aa58a267c0e671af97e P 067b2ec58c34466f99bd66c5780d8fd3: Neither blocks nor log segments found. Creating new log.
I20260806 06:26:55.478147 761 tablet_bootstrap.cc:654] T 40ad6b3dc98b4aa58a267c0e671af97e P 52f06bd6d6a448aeb07ada8c0d89e2e1: Neither blocks nor log segments found. Creating new log.
I20260806 06:26:55.478240 759 tablet_bootstrap.cc:492] T 40ad6b3dc98b4aa58a267c0e671af97e P 8483d5dffddc4ca080e13001a1707d12: No bootstrap required, opened a new log
I20260806 06:26:55.478332 759 ts_tablet_manager.cc:1397] T 40ad6b3dc98b4aa58a267c0e671af97e P 8483d5dffddc4ca080e13001a1707d12: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260806 06:26:55.478384 762 log.cc:824] T 40ad6b3dc98b4aa58a267c0e671af97e P 067b2ec58c34466f99bd66c5780d8fd3: Log is configured to *not* fsync() on all Append() calls
I20260806 06:26:55.478494 761 log.cc:824] T 40ad6b3dc98b4aa58a267c0e671af97e P 52f06bd6d6a448aeb07ada8c0d89e2e1: Log is configured to *not* fsync() on all Append() calls
I20260806 06:26:55.479135 762 tablet_bootstrap.cc:492] T 40ad6b3dc98b4aa58a267c0e671af97e P 067b2ec58c34466f99bd66c5780d8fd3: No bootstrap required, opened a new log
I20260806 06:26:55.479207 762 ts_tablet_manager.cc:1397] T 40ad6b3dc98b4aa58a267c0e671af97e P 067b2ec58c34466f99bd66c5780d8fd3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260806 06:26:55.479301 761 tablet_bootstrap.cc:492] T 40ad6b3dc98b4aa58a267c0e671af97e P 52f06bd6d6a448aeb07ada8c0d89e2e1: No bootstrap required, opened a new log
I20260806 06:26:55.479391 761 ts_tablet_manager.cc:1397] T 40ad6b3dc98b4aa58a267c0e671af97e P 52f06bd6d6a448aeb07ada8c0d89e2e1: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260806 06:26:55.480158 759 raft_consensus.cc:348] T 40ad6b3dc98b4aa58a267c0e671af97e P 8483d5dffddc4ca080e13001a1707d12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f06bd6d6a448aeb07ada8c0d89e2e1" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 39591 } } peers { permanent_uuid: "067b2ec58c34466f99bd66c5780d8fd3" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 44983 } } peers { permanent_uuid: "8483d5dffddc4ca080e13001a1707d12" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 41903 } }
I20260806 06:26:55.480286 759 raft_consensus.cc:374] T 40ad6b3dc98b4aa58a267c0e671af97e P 8483d5dffddc4ca080e13001a1707d12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 06:26:55.480316 759 raft_consensus.cc:729] T 40ad6b3dc98b4aa58a267c0e671af97e P 8483d5dffddc4ca080e13001a1707d12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8483d5dffddc4ca080e13001a1707d12, State: Initialized, Role: FOLLOWER
I20260806 06:26:55.480448 759 consensus_queue.cc:260] T 40ad6b3dc98b4aa58a267c0e671af97e P 8483d5dffddc4ca080e13001a1707d12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f06bd6d6a448aeb07ada8c0d89e2e1" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 39591 } } peers { permanent_uuid: "067b2ec58c34466f99bd66c5780d8fd3" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 44983 } } peers { permanent_uuid: "8483d5dffddc4ca080e13001a1707d12" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 41903 } }
I20260806 06:26:55.480679 759 ts_tablet_manager.cc:1428] T 40ad6b3dc98b4aa58a267c0e671af97e P 8483d5dffddc4ca080e13001a1707d12: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260806 06:26:55.480796 715 heartbeater.cc:503] Master 127.21.12.190:33805 was elected leader, sending a full tablet report...
I20260806 06:26:55.481072 762 raft_consensus.cc:348] T 40ad6b3dc98b4aa58a267c0e671af97e P 067b2ec58c34466f99bd66c5780d8fd3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f06bd6d6a448aeb07ada8c0d89e2e1" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 39591 } } peers { permanent_uuid: "067b2ec58c34466f99bd66c5780d8fd3" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 44983 } } peers { permanent_uuid: "8483d5dffddc4ca080e13001a1707d12" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 41903 } }
I20260806 06:26:55.481242 762 raft_consensus.cc:374] T 40ad6b3dc98b4aa58a267c0e671af97e P 067b2ec58c34466f99bd66c5780d8fd3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 06:26:55.481290 762 raft_consensus.cc:729] T 40ad6b3dc98b4aa58a267c0e671af97e P 067b2ec58c34466f99bd66c5780d8fd3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 067b2ec58c34466f99bd66c5780d8fd3, State: Initialized, Role: FOLLOWER
I20260806 06:26:55.481248 761 raft_consensus.cc:348] T 40ad6b3dc98b4aa58a267c0e671af97e P 52f06bd6d6a448aeb07ada8c0d89e2e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f06bd6d6a448aeb07ada8c0d89e2e1" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 39591 } } peers { permanent_uuid: "067b2ec58c34466f99bd66c5780d8fd3" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 44983 } } peers { permanent_uuid: "8483d5dffddc4ca080e13001a1707d12" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 41903 } }
I20260806 06:26:55.481381 761 raft_consensus.cc:374] T 40ad6b3dc98b4aa58a267c0e671af97e P 52f06bd6d6a448aeb07ada8c0d89e2e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 06:26:55.481413 761 raft_consensus.cc:729] T 40ad6b3dc98b4aa58a267c0e671af97e P 52f06bd6d6a448aeb07ada8c0d89e2e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52f06bd6d6a448aeb07ada8c0d89e2e1, State: Initialized, Role: FOLLOWER
I20260806 06:26:55.481400 762 consensus_queue.cc:260] T 40ad6b3dc98b4aa58a267c0e671af97e P 067b2ec58c34466f99bd66c5780d8fd3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f06bd6d6a448aeb07ada8c0d89e2e1" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 39591 } } peers { permanent_uuid: "067b2ec58c34466f99bd66c5780d8fd3" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 44983 } } peers { permanent_uuid: "8483d5dffddc4ca080e13001a1707d12" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 41903 } }
I20260806 06:26:55.481652 762 ts_tablet_manager.cc:1428] T 40ad6b3dc98b4aa58a267c0e671af97e P 067b2ec58c34466f99bd66c5780d8fd3: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260806 06:26:55.481600 761 consensus_queue.cc:260] T 40ad6b3dc98b4aa58a267c0e671af97e P 52f06bd6d6a448aeb07ada8c0d89e2e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f06bd6d6a448aeb07ada8c0d89e2e1" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 39591 } } peers { permanent_uuid: "067b2ec58c34466f99bd66c5780d8fd3" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 44983 } } peers { permanent_uuid: "8483d5dffddc4ca080e13001a1707d12" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 41903 } }
I20260806 06:26:55.481714 32761 heartbeater.cc:503] Master 127.21.12.190:33805 was elected leader, sending a full tablet report...
I20260806 06:26:55.481858 761 ts_tablet_manager.cc:1428] T 40ad6b3dc98b4aa58a267c0e671af97e P 52f06bd6d6a448aeb07ada8c0d89e2e1: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260806 06:26:55.482086 506 heartbeater.cc:503] Master 127.21.12.190:33805 was elected leader, sending a full tablet report...
W20260806 06:26:55.514168 508 tablet.cc:2369] T 40ad6b3dc98b4aa58a267c0e671af97e P 52f06bd6d6a448aeb07ada8c0d89e2e1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 06:26:55.528949 768 raft_consensus.cc:482] T 40ad6b3dc98b4aa58a267c0e671af97e P 067b2ec58c34466f99bd66c5780d8fd3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 06:26:55.529062 768 raft_consensus.cc:504] T 40ad6b3dc98b4aa58a267c0e671af97e P 067b2ec58c34466f99bd66c5780d8fd3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f06bd6d6a448aeb07ada8c0d89e2e1" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 39591 } } peers { permanent_uuid: "067b2ec58c34466f99bd66c5780d8fd3" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 44983 } } peers { permanent_uuid: "8483d5dffddc4ca080e13001a1707d12" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 41903 } }
I20260806 06:26:55.529404 768 leader_election.cc:288] T 40ad6b3dc98b4aa58a267c0e671af97e P 067b2ec58c34466f99bd66c5780d8fd3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52f06bd6d6a448aeb07ada8c0d89e2e1 (127.21.12.130:39591), 8483d5dffddc4ca080e13001a1707d12 (127.21.12.131:41903)
I20260806 06:26:55.532091 32595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:55.529565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.130:39591 (local address 127.21.12.129:59733)
0806 06:26:55.529621 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0806 06:26:55.529634 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 06:26:55.529691 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:26:55.529862 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:26:55.529864 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:26:55.529865 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 06:26:55.529932 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:55.529933 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:55.530556 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:55.530558 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:55.531327 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:55.531329 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:55.531840 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:55.531841 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:55.531918 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:55.531921 (+ 3us) client_negotiation.cc:769] Sending connection context
0806 06:26:55.531965 (+ 44us) client_negotiation.cc:240] Negotiation successful
0806 06:26:55.532005 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260806 06:26:55.532079 753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:55.529650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.129:59733 (local address 127.21.12.130:39591)
0806 06:26:55.529695 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0806 06:26:55.529697 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:26:55.529706 (+ 9us) server_negotiation.cc:407] Connection header received
0806 06:26:55.529778 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:26:55.529780 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:26:55.529791 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:26:55.529849 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 06:26:55.529988 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:55.530490 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:55.531413 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:55.531769 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:55.531828 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:55.531838 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0806 06:26:55.531980 (+ 142us) server_negotiation.cc:299] Negotiation successful
0806 06:26:55.532014 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260806 06:26:55.532195 757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:55.529772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.129:39925 (local address 127.21.12.131:41903)
0806 06:26:55.529818 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0806 06:26:55.529820 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:26:55.529923 (+ 103us) server_negotiation.cc:407] Connection header received
0806 06:26:55.529965 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:26:55.529966 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:26:55.529975 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:26:55.530023 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 06:26:55.530163 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:55.530653 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:55.531546 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:55.531887 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:55.531945 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:55.531953 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0806 06:26:55.532097 (+ 144us) server_negotiation.cc:299] Negotiation successful
0806 06:26:55.532130 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260806 06:26:55.532368 770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:55.529664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.131:41903 (local address 127.21.12.129:39925)
0806 06:26:55.529862 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0806 06:26:55.529871 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0806 06:26:55.529919 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:26:55.530033 (+ 114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:26:55.530033 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:26:55.530035 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 06:26:55.530106 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:55.530107 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:55.530731 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:55.530732 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:55.531462 (+ 730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:55.531464 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:55.531954 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:55.531955 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:55.532035 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:55.532038 (+ 3us) client_negotiation.cc:769] Sending connection context
0806 06:26:55.532082 (+ 44us) client_negotiation.cc:240] Negotiation successful
0806 06:26:55.532119 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":132,"threads_started":1}
I20260806 06:26:55.532398 436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40ad6b3dc98b4aa58a267c0e671af97e" candidate_uuid: "067b2ec58c34466f99bd66c5780d8fd3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52f06bd6d6a448aeb07ada8c0d89e2e1" is_pre_election: true
I20260806 06:26:55.532552 436 raft_consensus.cc:2454] T 40ad6b3dc98b4aa58a267c0e671af97e P 52f06bd6d6a448aeb07ada8c0d89e2e1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 067b2ec58c34466f99bd66c5780d8fd3 in term 0.
I20260806 06:26:55.532778 32589 leader_election.cc:302] T 40ad6b3dc98b4aa58a267c0e671af97e P 067b2ec58c34466f99bd66c5780d8fd3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 067b2ec58c34466f99bd66c5780d8fd3, 52f06bd6d6a448aeb07ada8c0d89e2e1; no voters:
I20260806 06:26:55.532859 648 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40ad6b3dc98b4aa58a267c0e671af97e" candidate_uuid: "067b2ec58c34466f99bd66c5780d8fd3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8483d5dffddc4ca080e13001a1707d12" is_pre_election: true
I20260806 06:26:55.532907 768 raft_consensus.cc:2781] T 40ad6b3dc98b4aa58a267c0e671af97e P 067b2ec58c34466f99bd66c5780d8fd3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 06:26:55.532968 768 raft_consensus.cc:482] T 40ad6b3dc98b4aa58a267c0e671af97e P 067b2ec58c34466f99bd66c5780d8fd3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 06:26:55.533000 768 raft_consensus.cc:3037] T 40ad6b3dc98b4aa58a267c0e671af97e P 067b2ec58c34466f99bd66c5780d8fd3 [term 0 FOLLOWER]: Advancing to term 1
I20260806 06:26:55.533034 648 raft_consensus.cc:2454] T 40ad6b3dc98b4aa58a267c0e671af97e P 8483d5dffddc4ca080e13001a1707d12 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 067b2ec58c34466f99bd66c5780d8fd3 in term 0.
I20260806 06:26:55.533939 768 raft_consensus.cc:504] T 40ad6b3dc98b4aa58a267c0e671af97e P 067b2ec58c34466f99bd66c5780d8fd3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f06bd6d6a448aeb07ada8c0d89e2e1" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 39591 } } peers { permanent_uuid: "067b2ec58c34466f99bd66c5780d8fd3" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 44983 } } peers { permanent_uuid: "8483d5dffddc4ca080e13001a1707d12" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 41903 } }
I20260806 06:26:55.534046 768 leader_election.cc:288] T 40ad6b3dc98b4aa58a267c0e671af97e P 067b2ec58c34466f99bd66c5780d8fd3 [CANDIDATE]: Term 1 election: Requested vote from peers 52f06bd6d6a448aeb07ada8c0d89e2e1 (127.21.12.130:39591), 8483d5dffddc4ca080e13001a1707d12 (127.21.12.131:41903)
I20260806 06:26:55.534209 436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40ad6b3dc98b4aa58a267c0e671af97e" candidate_uuid: "067b2ec58c34466f99bd66c5780d8fd3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52f06bd6d6a448aeb07ada8c0d89e2e1"
I20260806 06:26:55.534226 648 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40ad6b3dc98b4aa58a267c0e671af97e" candidate_uuid: "067b2ec58c34466f99bd66c5780d8fd3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8483d5dffddc4ca080e13001a1707d12"
I20260806 06:26:55.534278 436 raft_consensus.cc:3037] T 40ad6b3dc98b4aa58a267c0e671af97e P 52f06bd6d6a448aeb07ada8c0d89e2e1 [term 0 FOLLOWER]: Advancing to term 1
I20260806 06:26:55.534317 648 raft_consensus.cc:3037] T 40ad6b3dc98b4aa58a267c0e671af97e P 8483d5dffddc4ca080e13001a1707d12 [term 0 FOLLOWER]: Advancing to term 1
I20260806 06:26:55.534998 648 raft_consensus.cc:2454] T 40ad6b3dc98b4aa58a267c0e671af97e P 8483d5dffddc4ca080e13001a1707d12 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 067b2ec58c34466f99bd66c5780d8fd3 in term 1.
I20260806 06:26:55.535013 436 raft_consensus.cc:2454] T 40ad6b3dc98b4aa58a267c0e671af97e P 52f06bd6d6a448aeb07ada8c0d89e2e1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 067b2ec58c34466f99bd66c5780d8fd3 in term 1.
I20260806 06:26:55.535146 32589 leader_election.cc:302] T 40ad6b3dc98b4aa58a267c0e671af97e P 067b2ec58c34466f99bd66c5780d8fd3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 067b2ec58c34466f99bd66c5780d8fd3, 8483d5dffddc4ca080e13001a1707d12; no voters:
I20260806 06:26:55.535226 768 raft_consensus.cc:2781] T 40ad6b3dc98b4aa58a267c0e671af97e P 067b2ec58c34466f99bd66c5780d8fd3 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 06:26:55.535372 768 raft_consensus.cc:686] T 40ad6b3dc98b4aa58a267c0e671af97e P 067b2ec58c34466f99bd66c5780d8fd3 [term 1 LEADER]: Becoming Leader. State: Replica: 067b2ec58c34466f99bd66c5780d8fd3, State: Running, Role: LEADER
I20260806 06:26:55.535455 768 consensus_queue.cc:237] T 40ad6b3dc98b4aa58a267c0e671af97e P 067b2ec58c34466f99bd66c5780d8fd3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f06bd6d6a448aeb07ada8c0d89e2e1" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 39591 } } peers { permanent_uuid: "067b2ec58c34466f99bd66c5780d8fd3" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 44983 } } peers { permanent_uuid: "8483d5dffddc4ca080e13001a1707d12" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 41903 } }
I20260806 06:26:55.536268 32424 catalog_manager.cc:5495] T 40ad6b3dc98b4aa58a267c0e671af97e P 067b2ec58c34466f99bd66c5780d8fd3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 067b2ec58c34466f99bd66c5780d8fd3 (127.21.12.129). New cstate: current_term: 1 leader_uuid: "067b2ec58c34466f99bd66c5780d8fd3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f06bd6d6a448aeb07ada8c0d89e2e1" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 39591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "067b2ec58c34466f99bd66c5780d8fd3" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 44983 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8483d5dffddc4ca080e13001a1707d12" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 41903 } health_report { overall_health: UNKNOWN } } }
I20260806 06:26:55.543018 755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:55.540998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55214 (local address 127.21.12.129:44983)
0806 06:26:55.541047 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0806 06:26:55.541048 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0806 06:26:55.541055 (+ 7us) server_negotiation.cc:407] Connection header received
0806 06:26:55.541080 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:26:55.541081 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:26:55.541096 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:26:55.541154 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 06:26:55.541291 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:55.541740 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:55.542453 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:55.542654 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:55.542700 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:55.542800 (+ 100us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 06:26:55.542885 (+ 85us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 06:26:55.542940 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0806 06:26:55.542952 (+ 12us) server_negotiation.cc:299] Negotiation successful
0806 06:26:55.542967 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260806 06:26:55.544545 648 raft_consensus.cc:1261] T 40ad6b3dc98b4aa58a267c0e671af97e P 8483d5dffddc4ca080e13001a1707d12 [term 1 FOLLOWER]: Refusing update from remote peer 067b2ec58c34466f99bd66c5780d8fd3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 06:26:55.544572 436 raft_consensus.cc:1261] T 40ad6b3dc98b4aa58a267c0e671af97e P 52f06bd6d6a448aeb07ada8c0d89e2e1 [term 1 FOLLOWER]: Refusing update from remote peer 067b2ec58c34466f99bd66c5780d8fd3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 06:26:55.544843 768 consensus_queue.cc:1035] T 40ad6b3dc98b4aa58a267c0e671af97e P 067b2ec58c34466f99bd66c5780d8fd3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52f06bd6d6a448aeb07ada8c0d89e2e1" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 06:26:55.544973 772 consensus_queue.cc:1035] T 40ad6b3dc98b4aa58a267c0e671af97e P 067b2ec58c34466f99bd66c5780d8fd3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8483d5dffddc4ca080e13001a1707d12" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 41903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 06:26:55.547183 778 mvcc.cc:205] Tried to move back new op lower bound from 7315446233268285440 to 7315446233233825792. Current Snapshot: MvccSnapshot[applied={T|T < 7315446233268285440}]
I20260806 06:26:55.547323 780 mvcc.cc:205] Tried to move back new op lower bound from 7315446233268285440 to 7315446233233825792. Current Snapshot: MvccSnapshot[applied={T|T < 7315446233268285440}]
I20260806 06:26:55.549396 32424 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33140:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 06:26:55.549477 32424 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33140:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 06:26:55.550137 32424 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 987ca9a955c54e639f87808644890c4c: Sending DeleteTablet for 3 replicas of tablet 40ad6b3dc98b4aa58a267c0e671af97e
I20260806 06:26:55.550451 32651 tablet_service.cc:1514] Processing DeleteTablet for tablet 40ad6b3dc98b4aa58a267c0e671af97e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 06:26:55 UTC) from {username='oryx', principal='oryx/127.21.12.190@KRBTEST.COM'} at 127.0.0.1:55198
I20260806 06:26:55.550469 614 tablet_service.cc:1514] Processing DeleteTablet for tablet 40ad6b3dc98b4aa58a267c0e671af97e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 06:26:55 UTC) from {username='oryx', principal='oryx/127.21.12.190@KRBTEST.COM'} at 127.0.0.1:44776
I20260806 06:26:55.550519 404 tablet_service.cc:1514] Processing DeleteTablet for tablet 40ad6b3dc98b4aa58a267c0e671af97e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 06:26:55 UTC) from {username='oryx', principal='oryx/127.21.12.190@KRBTEST.COM'} at 127.0.0.1:39336
I20260806 06:26:55.550745 792 tablet_replica.cc:330] T 40ad6b3dc98b4aa58a267c0e671af97e P 8483d5dffddc4ca080e13001a1707d12: stopping tablet replica
I20260806 06:26:55.550752 32424 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33140:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260806 06:26:55.550848 792 raft_consensus.cc:2229] T 40ad6b3dc98b4aa58a267c0e671af97e P 8483d5dffddc4ca080e13001a1707d12 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260806 06:26:55.550894 32424 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 06:26:55.551187 792 raft_consensus.cc:2258] T 40ad6b3dc98b4aa58a267c0e671af97e P 8483d5dffddc4ca080e13001a1707d12 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 06:26:55.551908 792 ts_tablet_manager.cc:1905] T 40ad6b3dc98b4aa58a267c0e671af97e P 8483d5dffddc4ca080e13001a1707d12: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 06:26:55.552037 32651 tablet_service.cc:1467] Processing CreateTablet for tablet e698bde980464463a3b755b465be490d (DEFAULT_TABLE table=test-table [id=27460a7733f84b83873d847f33ac3698]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 06:26:55.552125 404 tablet_service.cc:1467] Processing CreateTablet for tablet e698bde980464463a3b755b465be490d (DEFAULT_TABLE table=test-table [id=27460a7733f84b83873d847f33ac3698]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 06:26:55.552153 32651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e698bde980464463a3b755b465be490d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 06:26:55.552212 404 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e698bde980464463a3b755b465be490d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 06:26:55.553445 762 tablet_bootstrap.cc:492] T e698bde980464463a3b755b465be490d P 067b2ec58c34466f99bd66c5780d8fd3: Bootstrap starting.
I20260806 06:26:55.553457 761 tablet_bootstrap.cc:492] T e698bde980464463a3b755b465be490d P 52f06bd6d6a448aeb07ada8c0d89e2e1: Bootstrap starting.
I20260806 06:26:55.553498 795 tablet_replica.cc:330] T 40ad6b3dc98b4aa58a267c0e671af97e P 52f06bd6d6a448aeb07ada8c0d89e2e1: stopping tablet replica
I20260806 06:26:55.553606 795 raft_consensus.cc:2229] T 40ad6b3dc98b4aa58a267c0e671af97e P 52f06bd6d6a448aeb07ada8c0d89e2e1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 06:26:55.553723 795 raft_consensus.cc:2258] T 40ad6b3dc98b4aa58a267c0e671af97e P 52f06bd6d6a448aeb07ada8c0d89e2e1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 06:26:55.553866 762 tablet_bootstrap.cc:654] T e698bde980464463a3b755b465be490d P 067b2ec58c34466f99bd66c5780d8fd3: Neither blocks nor log segments found. Creating new log.
I20260806 06:26:55.553882 792 ts_tablet_manager.cc:1918] T 40ad6b3dc98b4aa58a267c0e671af97e P 8483d5dffddc4ca080e13001a1707d12: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 06:26:55.553896 761 tablet_bootstrap.cc:654] T e698bde980464463a3b755b465be490d P 52f06bd6d6a448aeb07ada8c0d89e2e1: Neither blocks nor log segments found. Creating new log.
I20260806 06:26:55.553941 792 log.cc:1196] T 40ad6b3dc98b4aa58a267c0e671af97e P 8483d5dffddc4ca080e13001a1707d12: Deleting WAL directory at /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785997581049705-21554-0/minicluster-data/ts-2/wal/wals/40ad6b3dc98b4aa58a267c0e671af97e
I20260806 06:26:55.554201 792 ts_tablet_manager.cc:1939] T 40ad6b3dc98b4aa58a267c0e671af97e P 8483d5dffddc4ca080e13001a1707d12: Deleting consensus metadata
I20260806 06:26:55.554315 614 tablet_service.cc:1467] Processing CreateTablet for tablet e698bde980464463a3b755b465be490d (DEFAULT_TABLE table=test-table [id=27460a7733f84b83873d847f33ac3698]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 06:26:55.554433 614 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e698bde980464463a3b755b465be490d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 06:26:55.554661 761 tablet_bootstrap.cc:492] T e698bde980464463a3b755b465be490d P 52f06bd6d6a448aeb07ada8c0d89e2e1: No bootstrap required, opened a new log
I20260806 06:26:55.554703 761 ts_tablet_manager.cc:1397] T e698bde980464463a3b755b465be490d P 52f06bd6d6a448aeb07ada8c0d89e2e1: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260806 06:26:55.554836 761 raft_consensus.cc:348] T e698bde980464463a3b755b465be490d P 52f06bd6d6a448aeb07ada8c0d89e2e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "067b2ec58c34466f99bd66c5780d8fd3" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 44983 } } peers { permanent_uuid: "52f06bd6d6a448aeb07ada8c0d89e2e1" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 39591 } } peers { permanent_uuid: "8483d5dffddc4ca080e13001a1707d12" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 41903 } }
I20260806 06:26:55.554906 761 raft_consensus.cc:374] T e698bde980464463a3b755b465be490d P 52f06bd6d6a448aeb07ada8c0d89e2e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 06:26:55.554925 761 raft_consensus.cc:729] T e698bde980464463a3b755b465be490d P 52f06bd6d6a448aeb07ada8c0d89e2e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52f06bd6d6a448aeb07ada8c0d89e2e1, State: Initialized, Role: FOLLOWER
I20260806 06:26:55.554944 795 ts_tablet_manager.cc:1905] T 40ad6b3dc98b4aa58a267c0e671af97e P 52f06bd6d6a448aeb07ada8c0d89e2e1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 06:26:55.554967 761 consensus_queue.cc:260] T e698bde980464463a3b755b465be490d P 52f06bd6d6a448aeb07ada8c0d89e2e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "067b2ec58c34466f99bd66c5780d8fd3" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 44983 } } peers { permanent_uuid: "52f06bd6d6a448aeb07ada8c0d89e2e1" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 39591 } } peers { permanent_uuid: "8483d5dffddc4ca080e13001a1707d12" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 41903 } }
I20260806 06:26:55.555048 761 ts_tablet_manager.cc:1428] T e698bde980464463a3b755b465be490d P 52f06bd6d6a448aeb07ada8c0d89e2e1: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260806 06:26:55.555589 759 tablet_bootstrap.cc:492] T e698bde980464463a3b755b465be490d P 8483d5dffddc4ca080e13001a1707d12: Bootstrap starting.
I20260806 06:26:55.555642 32406 catalog_manager.cc:4841] TS 8483d5dffddc4ca080e13001a1707d12 (127.21.12.131:41903): tablet 40ad6b3dc98b4aa58a267c0e671af97e (table test-table [id=5fe9e3990d3e41d6ae29f70066ff9dea]) successfully deleted
I20260806 06:26:55.555992 759 tablet_bootstrap.cc:654] T e698bde980464463a3b755b465be490d P 8483d5dffddc4ca080e13001a1707d12: Neither blocks nor log segments found. Creating new log.
I20260806 06:26:55.556553 759 tablet_bootstrap.cc:492] T e698bde980464463a3b755b465be490d P 8483d5dffddc4ca080e13001a1707d12: No bootstrap required, opened a new log
I20260806 06:26:55.556596 759 ts_tablet_manager.cc:1397] T e698bde980464463a3b755b465be490d P 8483d5dffddc4ca080e13001a1707d12: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260806 06:26:55.556651 795 ts_tablet_manager.cc:1918] T 40ad6b3dc98b4aa58a267c0e671af97e P 52f06bd6d6a448aeb07ada8c0d89e2e1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 06:26:55.556725 795 log.cc:1196] T 40ad6b3dc98b4aa58a267c0e671af97e P 52f06bd6d6a448aeb07ada8c0d89e2e1: Deleting WAL directory at /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785997581049705-21554-0/minicluster-data/ts-1/wal/wals/40ad6b3dc98b4aa58a267c0e671af97e
I20260806 06:26:55.556730 759 raft_consensus.cc:348] T e698bde980464463a3b755b465be490d P 8483d5dffddc4ca080e13001a1707d12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "067b2ec58c34466f99bd66c5780d8fd3" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 44983 } } peers { permanent_uuid: "52f06bd6d6a448aeb07ada8c0d89e2e1" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 39591 } } peers { permanent_uuid: "8483d5dffddc4ca080e13001a1707d12" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 41903 } }
I20260806 06:26:55.556797 759 raft_consensus.cc:374] T e698bde980464463a3b755b465be490d P 8483d5dffddc4ca080e13001a1707d12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 06:26:55.556820 759 raft_consensus.cc:729] T e698bde980464463a3b755b465be490d P 8483d5dffddc4ca080e13001a1707d12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8483d5dffddc4ca080e13001a1707d12, State: Initialized, Role: FOLLOWER
I20260806 06:26:55.556864 759 consensus_queue.cc:260] T e698bde980464463a3b755b465be490d P 8483d5dffddc4ca080e13001a1707d12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "067b2ec58c34466f99bd66c5780d8fd3" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 44983 } } peers { permanent_uuid: "52f06bd6d6a448aeb07ada8c0d89e2e1" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 39591 } } peers { permanent_uuid: "8483d5dffddc4ca080e13001a1707d12" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 41903 } }
I20260806 06:26:55.556945 759 ts_tablet_manager.cc:1428] T e698bde980464463a3b755b465be490d P 8483d5dffddc4ca080e13001a1707d12: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260806 06:26:55.557004 795 ts_tablet_manager.cc:1939] T 40ad6b3dc98b4aa58a267c0e671af97e P 52f06bd6d6a448aeb07ada8c0d89e2e1: Deleting consensus metadata
I20260806 06:26:55.557441 32406 catalog_manager.cc:4841] TS 52f06bd6d6a448aeb07ada8c0d89e2e1 (127.21.12.130:39591): tablet 40ad6b3dc98b4aa58a267c0e671af97e (table test-table [id=5fe9e3990d3e41d6ae29f70066ff9dea]) successfully deleted
I20260806 06:26:55.558410 791 tablet_replica.cc:330] T 40ad6b3dc98b4aa58a267c0e671af97e P 067b2ec58c34466f99bd66c5780d8fd3: stopping tablet replica
I20260806 06:26:55.558553 791 raft_consensus.cc:2229] T 40ad6b3dc98b4aa58a267c0e671af97e P 067b2ec58c34466f99bd66c5780d8fd3 [term 1 LEADER]: Raft consensus shutting down.
I20260806 06:26:55.558678 791 raft_consensus.cc:2258] T 40ad6b3dc98b4aa58a267c0e671af97e P 067b2ec58c34466f99bd66c5780d8fd3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 06:26:55.559119 762 tablet_bootstrap.cc:492] T e698bde980464463a3b755b465be490d P 067b2ec58c34466f99bd66c5780d8fd3: No bootstrap required, opened a new log
I20260806 06:26:55.559150 791 ts_tablet_manager.cc:1905] T 40ad6b3dc98b4aa58a267c0e671af97e P 067b2ec58c34466f99bd66c5780d8fd3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 06:26:55.559161 762 ts_tablet_manager.cc:1397] T e698bde980464463a3b755b465be490d P 067b2ec58c34466f99bd66c5780d8fd3: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20260806 06:26:55.559329 762 raft_consensus.cc:348] T e698bde980464463a3b755b465be490d P 067b2ec58c34466f99bd66c5780d8fd3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "067b2ec58c34466f99bd66c5780d8fd3" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 44983 } } peers { permanent_uuid: "52f06bd6d6a448aeb07ada8c0d89e2e1" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 39591 } } peers { permanent_uuid: "8483d5dffddc4ca080e13001a1707d12" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 41903 } }
I20260806 06:26:55.559381 762 raft_consensus.cc:374] T e698bde980464463a3b755b465be490d P 067b2ec58c34466f99bd66c5780d8fd3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 06:26:55.559397 762 raft_consensus.cc:729] T e698bde980464463a3b755b465be490d P 067b2ec58c34466f99bd66c5780d8fd3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 067b2ec58c34466f99bd66c5780d8fd3, State: Initialized, Role: FOLLOWER
I20260806 06:26:55.559440 762 consensus_queue.cc:260] T e698bde980464463a3b755b465be490d P 067b2ec58c34466f99bd66c5780d8fd3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "067b2ec58c34466f99bd66c5780d8fd3" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 44983 } } peers { permanent_uuid: "52f06bd6d6a448aeb07ada8c0d89e2e1" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 39591 } } peers { permanent_uuid: "8483d5dffddc4ca080e13001a1707d12" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 41903 } }
I20260806 06:26:55.559522 762 ts_tablet_manager.cc:1428] T e698bde980464463a3b755b465be490d P 067b2ec58c34466f99bd66c5780d8fd3: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260806 06:26:55.560684 791 ts_tablet_manager.cc:1918] T 40ad6b3dc98b4aa58a267c0e671af97e P 067b2ec58c34466f99bd66c5780d8fd3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 06:26:55.560762 791 log.cc:1196] T 40ad6b3dc98b4aa58a267c0e671af97e P 067b2ec58c34466f99bd66c5780d8fd3: Deleting WAL directory at /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785997581049705-21554-0/minicluster-data/ts-0/wal/wals/40ad6b3dc98b4aa58a267c0e671af97e
I20260806 06:26:55.560995 791 ts_tablet_manager.cc:1939] T 40ad6b3dc98b4aa58a267c0e671af97e P 067b2ec58c34466f99bd66c5780d8fd3: Deleting consensus metadata
I20260806 06:26:55.561339 32407 catalog_manager.cc:4841] TS 067b2ec58c34466f99bd66c5780d8fd3 (127.21.12.129:44983): tablet 40ad6b3dc98b4aa58a267c0e671af97e (table test-table [id=5fe9e3990d3e41d6ae29f70066ff9dea]) successfully deleted
W20260806 06:26:55.619288 32762 tablet.cc:2369] T e698bde980464463a3b755b465be490d P 067b2ec58c34466f99bd66c5780d8fd3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 06:26:55.672250 716 tablet.cc:2369] T e698bde980464463a3b755b465be490d P 8483d5dffddc4ca080e13001a1707d12: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 06:26:55.822723 767 raft_consensus.cc:482] T e698bde980464463a3b755b465be490d P 8483d5dffddc4ca080e13001a1707d12 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 06:26:55.822825 767 raft_consensus.cc:504] T e698bde980464463a3b755b465be490d P 8483d5dffddc4ca080e13001a1707d12 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "067b2ec58c34466f99bd66c5780d8fd3" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 44983 } } peers { permanent_uuid: "52f06bd6d6a448aeb07ada8c0d89e2e1" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 39591 } } peers { permanent_uuid: "8483d5dffddc4ca080e13001a1707d12" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 41903 } }
I20260806 06:26:55.823168 767 leader_election.cc:288] T e698bde980464463a3b755b465be490d P 8483d5dffddc4ca080e13001a1707d12 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 067b2ec58c34466f99bd66c5780d8fd3 (127.21.12.129:44983), 52f06bd6d6a448aeb07ada8c0d89e2e1 (127.21.12.130:39591)
I20260806 06:26:55.826072 755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:55.823409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.131:55091 (local address 127.21.12.129:44983)
0806 06:26:55.823466 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0806 06:26:55.823468 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:26:55.823625 (+ 157us) server_negotiation.cc:407] Connection header received
0806 06:26:55.823675 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:26:55.823676 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:26:55.823689 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:26:55.823742 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 06:26:55.823899 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:55.824370 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:55.825393 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:55.825761 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:55.825823 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:55.825832 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0806 06:26:55.825973 (+ 141us) server_negotiation.cc:299] Negotiation successful
0806 06:26:55.826006 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260806 06:26:55.826074 753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:55.823478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.131:41237 (local address 127.21.12.130:39591)
0806 06:26:55.823529 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0806 06:26:55.823532 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 06:26:55.823541 (+ 9us) server_negotiation.cc:407] Connection header received
0806 06:26:55.823572 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:26:55.823573 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:26:55.823584 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:26:55.823637 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 06:26:55.823814 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:55.824357 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:55.825407 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:55.825758 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:55.825816 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:55.825826 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0806 06:26:55.825964 (+ 138us) server_negotiation.cc:299] Negotiation successful
0806 06:26:55.826011 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260806 06:26:55.826082 560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:55.823348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.130:39591 (local address 127.21.12.131:41237)
0806 06:26:55.823395 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0806 06:26:55.823407 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 06:26:55.823455 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:26:55.823660 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:26:55.823661 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:26:55.823664 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 06:26:55.823744 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:55.823747 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:55.824423 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:55.824424 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:55.825319 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:55.825322 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:55.825826 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:55.825827 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:55.825905 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:55.825908 (+ 3us) client_negotiation.cc:769] Sending connection context
0806 06:26:55.825951 (+ 43us) client_negotiation.cc:240] Negotiation successful
0806 06:26:55.825990 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23,"mutex_wait_us":27}
I20260806 06:26:55.826095 799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:55.823354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.129:44983 (local address 127.21.12.131:55091)
0806 06:26:55.823559 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0806 06:26:55.823567 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0806 06:26:55.823615 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:26:55.823749 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:26:55.823750 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:26:55.823751 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 06:26:55.823839 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:55.823841 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:55.824434 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:55.824435 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:55.825325 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:55.825328 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:55.825849 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:55.825850 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:55.825911 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:55.825914 (+ 3us) client_negotiation.cc:769] Sending connection context
0806 06:26:55.825961 (+ 47us) client_negotiation.cc:240] Negotiation successful
0806 06:26:55.826000 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":195,"mutex_wait_us":84,"thread_start_us":65,"threads_started":1}
I20260806 06:26:55.826401 436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e698bde980464463a3b755b465be490d" candidate_uuid: "8483d5dffddc4ca080e13001a1707d12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52f06bd6d6a448aeb07ada8c0d89e2e1" is_pre_election: true
I20260806 06:26:55.826478 436 raft_consensus.cc:2454] T e698bde980464463a3b755b465be490d P 52f06bd6d6a448aeb07ada8c0d89e2e1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8483d5dffddc4ca080e13001a1707d12 in term 0.
I20260806 06:26:55.826583 32683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e698bde980464463a3b755b465be490d" candidate_uuid: "8483d5dffddc4ca080e13001a1707d12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "067b2ec58c34466f99bd66c5780d8fd3" is_pre_election: true
I20260806 06:26:55.826649 551 leader_election.cc:302] T e698bde980464463a3b755b465be490d P 8483d5dffddc4ca080e13001a1707d12 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52f06bd6d6a448aeb07ada8c0d89e2e1, 8483d5dffddc4ca080e13001a1707d12; no voters:
I20260806 06:26:55.826717 32683 raft_consensus.cc:2454] T e698bde980464463a3b755b465be490d P 067b2ec58c34466f99bd66c5780d8fd3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8483d5dffddc4ca080e13001a1707d12 in term 0.
I20260806 06:26:55.826772 767 raft_consensus.cc:2781] T e698bde980464463a3b755b465be490d P 8483d5dffddc4ca080e13001a1707d12 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 06:26:55.826817 767 raft_consensus.cc:482] T e698bde980464463a3b755b465be490d P 8483d5dffddc4ca080e13001a1707d12 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 06:26:55.826836 767 raft_consensus.cc:3037] T e698bde980464463a3b755b465be490d P 8483d5dffddc4ca080e13001a1707d12 [term 0 FOLLOWER]: Advancing to term 1
I20260806 06:26:55.827509 767 raft_consensus.cc:504] T e698bde980464463a3b755b465be490d P 8483d5dffddc4ca080e13001a1707d12 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "067b2ec58c34466f99bd66c5780d8fd3" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 44983 } } peers { permanent_uuid: "52f06bd6d6a448aeb07ada8c0d89e2e1" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 39591 } } peers { permanent_uuid: "8483d5dffddc4ca080e13001a1707d12" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 41903 } }
I20260806 06:26:55.827621 767 leader_election.cc:288] T e698bde980464463a3b755b465be490d P 8483d5dffddc4ca080e13001a1707d12 [CANDIDATE]: Term 1 election: Requested vote from peers 067b2ec58c34466f99bd66c5780d8fd3 (127.21.12.129:44983), 52f06bd6d6a448aeb07ada8c0d89e2e1 (127.21.12.130:39591)
I20260806 06:26:55.827773 32683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e698bde980464463a3b755b465be490d" candidate_uuid: "8483d5dffddc4ca080e13001a1707d12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "067b2ec58c34466f99bd66c5780d8fd3"
I20260806 06:26:55.827773 436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e698bde980464463a3b755b465be490d" candidate_uuid: "8483d5dffddc4ca080e13001a1707d12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52f06bd6d6a448aeb07ada8c0d89e2e1"
I20260806 06:26:55.827853 436 raft_consensus.cc:3037] T e698bde980464463a3b755b465be490d P 52f06bd6d6a448aeb07ada8c0d89e2e1 [term 0 FOLLOWER]: Advancing to term 1
I20260806 06:26:55.827853 32683 raft_consensus.cc:3037] T e698bde980464463a3b755b465be490d P 067b2ec58c34466f99bd66c5780d8fd3 [term 0 FOLLOWER]: Advancing to term 1
I20260806 06:26:55.828583 436 raft_consensus.cc:2454] T e698bde980464463a3b755b465be490d P 52f06bd6d6a448aeb07ada8c0d89e2e1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8483d5dffddc4ca080e13001a1707d12 in term 1.
I20260806 06:26:55.828629 32683 raft_consensus.cc:2454] T e698bde980464463a3b755b465be490d P 067b2ec58c34466f99bd66c5780d8fd3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8483d5dffddc4ca080e13001a1707d12 in term 1.
I20260806 06:26:55.828771 552 leader_election.cc:302] T e698bde980464463a3b755b465be490d P 8483d5dffddc4ca080e13001a1707d12 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 067b2ec58c34466f99bd66c5780d8fd3, 8483d5dffddc4ca080e13001a1707d12; no voters:
I20260806 06:26:55.828869 767 raft_consensus.cc:2781] T e698bde980464463a3b755b465be490d P 8483d5dffddc4ca080e13001a1707d12 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 06:26:55.829010 767 raft_consensus.cc:686] T e698bde980464463a3b755b465be490d P 8483d5dffddc4ca080e13001a1707d12 [term 1 LEADER]: Becoming Leader. State: Replica: 8483d5dffddc4ca080e13001a1707d12, State: Running, Role: LEADER
I20260806 06:26:55.829073 767 consensus_queue.cc:237] T e698bde980464463a3b755b465be490d P 8483d5dffddc4ca080e13001a1707d12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "067b2ec58c34466f99bd66c5780d8fd3" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 44983 } } peers { permanent_uuid: "52f06bd6d6a448aeb07ada8c0d89e2e1" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 39591 } } peers { permanent_uuid: "8483d5dffddc4ca080e13001a1707d12" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 41903 } }
I20260806 06:26:55.829771 32424 catalog_manager.cc:5495] T e698bde980464463a3b755b465be490d P 8483d5dffddc4ca080e13001a1707d12 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8483d5dffddc4ca080e13001a1707d12 (127.21.12.131). New cstate: current_term: 1 leader_uuid: "8483d5dffddc4ca080e13001a1707d12" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "067b2ec58c34466f99bd66c5780d8fd3" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 44983 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52f06bd6d6a448aeb07ada8c0d89e2e1" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 39591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8483d5dffddc4ca080e13001a1707d12" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 41903 } health_report { overall_health: HEALTHY } } }
Aug 06 06:26:55 dist-test-slave-4tdj krb5kdc[32356](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997614, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.12.190@KRBTEST.COM for oryx/127.21.12.190@KRBTEST.COM
I20260806 06:26:55.873824 812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:55.867114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:33805 (local address 127.0.0.1:33152)
0806 06:26:55.867307 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0806 06:26:55.867317 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0806 06:26:55.867390 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:26:55.867521 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:26:55.867521 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:26:55.867840 (+ 319us) client_negotiation.cc:189] Negotiated authn=SASL
0806 06:26:55.867900 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:55.867903 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:55.868549 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:55.868550 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:55.869084 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:55.869085 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:55.869388 (+ 303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:55.869388 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:55.869436 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:55.869909 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 06:26:55.869913 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0806 06:26:55.871477 (+ 1564us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 06:26:55.872836 (+ 1359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:26:55.872845 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:26:55.872848 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:26:55.873040 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:26:55.873209 (+ 169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:26:55.873209 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:26:55.873210 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:26:55.873261 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:26:55.873460 (+ 199us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 06:26:55.873468 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 06:26:55.873516 (+ 48us) client_negotiation.cc:769] Sending connection context
0806 06:26:55.873580 (+ 64us) client_negotiation.cc:240] Negotiation successful
0806 06:26:55.873708 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"mutex_wait_us":2,"thread_start_us":80,"threads_started":1}
I20260806 06:26:55.873871 32549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:55.867160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33152 (local address 127.21.12.190:33805)
0806 06:26:55.867234 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0806 06:26:55.867237 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 06:26:55.867384 (+ 147us) server_negotiation.cc:407] Connection header received
0806 06:26:55.867444 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:26:55.867445 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:26:55.867455 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:26:55.867517 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:26:55.867981 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:55.868462 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:55.869159 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:55.869332 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:55.869384 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:55.871585 (+ 2201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:26:55.871587 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:26:55.871587 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 06:26:55.871594 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:26:55.872756 (+ 1162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:26:55.873104 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:26:55.873105 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:26:55.873106 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:26:55.873156 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:26:55.873308 (+ 152us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:26:55.873309 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:26:55.873309 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:26:55.873409 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:26:55.873463 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0806 06:26:55.873609 (+ 146us) server_negotiation.cc:299] Negotiation successful
0806 06:26:55.873770 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":90,"server-negotiator.queue_time_us":45}
I20260806 06:26:55.875780 32424 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.21.12.190@KRBTEST.COM'} at 127.0.0.1:33152:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260806 06:26:55.876001 32424 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 06:26:55.877374 32651 tablet_service.cc:1467] Processing CreateTablet for tablet d4315b4746e84442815c3de0e6c4984e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=8f60afc29ecd438f9135a963e83d5d04]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 06:26:55.877403 404 tablet_service.cc:1467] Processing CreateTablet for tablet d4315b4746e84442815c3de0e6c4984e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=8f60afc29ecd438f9135a963e83d5d04]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 06:26:55.877453 614 tablet_service.cc:1467] Processing CreateTablet for tablet d4315b4746e84442815c3de0e6c4984e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=8f60afc29ecd438f9135a963e83d5d04]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 06:26:55.877552 404 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4315b4746e84442815c3de0e6c4984e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 06:26:55.877557 32651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4315b4746e84442815c3de0e6c4984e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 06:26:55.877630 614 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4315b4746e84442815c3de0e6c4984e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 06:26:55.878918 759 tablet_bootstrap.cc:492] T d4315b4746e84442815c3de0e6c4984e P 8483d5dffddc4ca080e13001a1707d12: Bootstrap starting.
I20260806 06:26:55.879040 762 tablet_bootstrap.cc:492] T d4315b4746e84442815c3de0e6c4984e P 067b2ec58c34466f99bd66c5780d8fd3: Bootstrap starting.
I20260806 06:26:55.879352 759 tablet_bootstrap.cc:654] T d4315b4746e84442815c3de0e6c4984e P 8483d5dffddc4ca080e13001a1707d12: Neither blocks nor log segments found. Creating new log.
I20260806 06:26:55.879360 761 tablet_bootstrap.cc:492] T d4315b4746e84442815c3de0e6c4984e P 52f06bd6d6a448aeb07ada8c0d89e2e1: Bootstrap starting.
I20260806 06:26:55.879379 762 tablet_bootstrap.cc:654] T d4315b4746e84442815c3de0e6c4984e P 067b2ec58c34466f99bd66c5780d8fd3: Neither blocks nor log segments found. Creating new log.
I20260806 06:26:55.879760 761 tablet_bootstrap.cc:654] T d4315b4746e84442815c3de0e6c4984e P 52f06bd6d6a448aeb07ada8c0d89e2e1: Neither blocks nor log segments found. Creating new log.
I20260806 06:26:55.879935 762 tablet_bootstrap.cc:492] T d4315b4746e84442815c3de0e6c4984e P 067b2ec58c34466f99bd66c5780d8fd3: No bootstrap required, opened a new log
I20260806 06:26:55.879983 762 ts_tablet_manager.cc:1397] T d4315b4746e84442815c3de0e6c4984e P 067b2ec58c34466f99bd66c5780d8fd3: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260806 06:26:55.880023 759 tablet_bootstrap.cc:492] T d4315b4746e84442815c3de0e6c4984e P 8483d5dffddc4ca080e13001a1707d12: No bootstrap required, opened a new log
I20260806 06:26:55.880075 759 ts_tablet_manager.cc:1397] T d4315b4746e84442815c3de0e6c4984e P 8483d5dffddc4ca080e13001a1707d12: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260806 06:26:55.880100 762 raft_consensus.cc:348] T d4315b4746e84442815c3de0e6c4984e P 067b2ec58c34466f99bd66c5780d8fd3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "067b2ec58c34466f99bd66c5780d8fd3" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 44983 } } peers { permanent_uuid: "52f06bd6d6a448aeb07ada8c0d89e2e1" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 39591 } } peers { permanent_uuid: "8483d5dffddc4ca080e13001a1707d12" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 41903 } }
I20260806 06:26:55.880167 762 raft_consensus.cc:374] T d4315b4746e84442815c3de0e6c4984e P 067b2ec58c34466f99bd66c5780d8fd3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 06:26:55.880185 762 raft_consensus.cc:729] T d4315b4746e84442815c3de0e6c4984e P 067b2ec58c34466f99bd66c5780d8fd3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 067b2ec58c34466f99bd66c5780d8fd3, State: Initialized, Role: FOLLOWER
I20260806 06:26:55.880226 762 consensus_queue.cc:260] T d4315b4746e84442815c3de0e6c4984e P 067b2ec58c34466f99bd66c5780d8fd3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "067b2ec58c34466f99bd66c5780d8fd3" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 44983 } } peers { permanent_uuid: "52f06bd6d6a448aeb07ada8c0d89e2e1" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 39591 } } peers { permanent_uuid: "8483d5dffddc4ca080e13001a1707d12" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 41903 } }
I20260806 06:26:55.880316 762 ts_tablet_manager.cc:1428] T d4315b4746e84442815c3de0e6c4984e P 067b2ec58c34466f99bd66c5780d8fd3: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260806 06:26:55.880358 761 tablet_bootstrap.cc:492] T d4315b4746e84442815c3de0e6c4984e P 52f06bd6d6a448aeb07ada8c0d89e2e1: No bootstrap required, opened a new log
I20260806 06:26:55.880396 761 ts_tablet_manager.cc:1397] T d4315b4746e84442815c3de0e6c4984e P 52f06bd6d6a448aeb07ada8c0d89e2e1: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260806 06:26:55.880354 759 raft_consensus.cc:348] T d4315b4746e84442815c3de0e6c4984e P 8483d5dffddc4ca080e13001a1707d12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "067b2ec58c34466f99bd66c5780d8fd3" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 44983 } } peers { permanent_uuid: "52f06bd6d6a448aeb07ada8c0d89e2e1" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 39591 } } peers { permanent_uuid: "8483d5dffddc4ca080e13001a1707d12" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 41903 } }
I20260806 06:26:55.880370 768 tablet_replica.cc:439] T d4315b4746e84442815c3de0e6c4984e P 067b2ec58c34466f99bd66c5780d8fd3: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "067b2ec58c34466f99bd66c5780d8fd3" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 44983 } } peers { permanent_uuid: "52f06bd6d6a448aeb07ada8c0d89e2e1" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 39591 } } peers { permanent_uuid: "8483d5dffddc4ca080e13001a1707d12" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 41903 } } }
I20260806 06:26:55.880436 768 tablet_replica.cc:442] T d4315b4746e84442815c3de0e6c4984e P 067b2ec58c34466f99bd66c5780d8fd3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 06:26:55.880425 759 raft_consensus.cc:374] T d4315b4746e84442815c3de0e6c4984e P 8483d5dffddc4ca080e13001a1707d12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 06:26:55.880479 759 raft_consensus.cc:729] T d4315b4746e84442815c3de0e6c4984e P 8483d5dffddc4ca080e13001a1707d12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8483d5dffddc4ca080e13001a1707d12, State: Initialized, Role: FOLLOWER
I20260806 06:26:55.880519 759 consensus_queue.cc:260] T d4315b4746e84442815c3de0e6c4984e P 8483d5dffddc4ca080e13001a1707d12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "067b2ec58c34466f99bd66c5780d8fd3" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 44983 } } peers { permanent_uuid: "52f06bd6d6a448aeb07ada8c0d89e2e1" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 39591 } } peers { permanent_uuid: "8483d5dffddc4ca080e13001a1707d12" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 41903 } }
I20260806 06:26:55.880549 761 raft_consensus.cc:348] T d4315b4746e84442815c3de0e6c4984e P 52f06bd6d6a448aeb07ada8c0d89e2e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "067b2ec58c34466f99bd66c5780d8fd3" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 44983 } } peers { permanent_uuid: "52f06bd6d6a448aeb07ada8c0d89e2e1" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 39591 } } peers { permanent_uuid: "8483d5dffddc4ca080e13001a1707d12" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 41903 } }
I20260806 06:26:55.880605 759 ts_tablet_manager.cc:1428] T d4315b4746e84442815c3de0e6c4984e P 8483d5dffddc4ca080e13001a1707d12: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 06:26:55.880611 761 raft_consensus.cc:374] T d4315b4746e84442815c3de0e6c4984e P 52f06bd6d6a448aeb07ada8c0d89e2e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 06:26:55.880630 761 raft_consensus.cc:729] T d4315b4746e84442815c3de0e6c4984e P 52f06bd6d6a448aeb07ada8c0d89e2e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52f06bd6d6a448aeb07ada8c0d89e2e1, State: Initialized, Role: FOLLOWER
I20260806 06:26:55.880669 761 consensus_queue.cc:260] T d4315b4746e84442815c3de0e6c4984e P 52f06bd6d6a448aeb07ada8c0d89e2e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "067b2ec58c34466f99bd66c5780d8fd3" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 44983 } } peers { permanent_uuid: "52f06bd6d6a448aeb07ada8c0d89e2e1" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 39591 } } peers { permanent_uuid: "8483d5dffddc4ca080e13001a1707d12" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 41903 } }
I20260806 06:26:55.880702 767 tablet_replica.cc:439] T d4315b4746e84442815c3de0e6c4984e P 8483d5dffddc4ca080e13001a1707d12: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "067b2ec58c34466f99bd66c5780d8fd3" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 44983 } } peers { permanent_uuid: "52f06bd6d6a448aeb07ada8c0d89e2e1" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 39591 } } peers { permanent_uuid: "8483d5dffddc4ca080e13001a1707d12" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 41903 } } }
I20260806 06:26:55.880749 761 ts_tablet_manager.cc:1428] T d4315b4746e84442815c3de0e6c4984e P 52f06bd6d6a448aeb07ada8c0d89e2e1: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260806 06:26:55.880759 767 tablet_replica.cc:442] T d4315b4746e84442815c3de0e6c4984e P 8483d5dffddc4ca080e13001a1707d12: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 06:26:55.880873 769 tablet_replica.cc:439] T d4315b4746e84442815c3de0e6c4984e P 52f06bd6d6a448aeb07ada8c0d89e2e1: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "067b2ec58c34466f99bd66c5780d8fd3" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 44983 } } peers { permanent_uuid: "52f06bd6d6a448aeb07ada8c0d89e2e1" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 39591 } } peers { permanent_uuid: "8483d5dffddc4ca080e13001a1707d12" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 41903 } } }
I20260806 06:26:55.880959 769 tablet_replica.cc:442] T d4315b4746e84442815c3de0e6c4984e P 52f06bd6d6a448aeb07ada8c0d89e2e1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 06:26:55.938534 767 raft_consensus.cc:482] T d4315b4746e84442815c3de0e6c4984e P 8483d5dffddc4ca080e13001a1707d12 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 06:26:55.938656 767 raft_consensus.cc:504] T d4315b4746e84442815c3de0e6c4984e P 8483d5dffddc4ca080e13001a1707d12 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "067b2ec58c34466f99bd66c5780d8fd3" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 44983 } } peers { permanent_uuid: "52f06bd6d6a448aeb07ada8c0d89e2e1" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 39591 } } peers { permanent_uuid: "8483d5dffddc4ca080e13001a1707d12" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 41903 } }
I20260806 06:26:55.938820 767 leader_election.cc:288] T d4315b4746e84442815c3de0e6c4984e P 8483d5dffddc4ca080e13001a1707d12 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 067b2ec58c34466f99bd66c5780d8fd3 (127.21.12.129:44983), 52f06bd6d6a448aeb07ada8c0d89e2e1 (127.21.12.130:39591)
I20260806 06:26:55.939038 436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4315b4746e84442815c3de0e6c4984e" candidate_uuid: "8483d5dffddc4ca080e13001a1707d12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52f06bd6d6a448aeb07ada8c0d89e2e1" is_pre_election: true
I20260806 06:26:55.939038 32683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4315b4746e84442815c3de0e6c4984e" candidate_uuid: "8483d5dffddc4ca080e13001a1707d12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "067b2ec58c34466f99bd66c5780d8fd3" is_pre_election: true
I20260806 06:26:55.939157 436 raft_consensus.cc:2454] T d4315b4746e84442815c3de0e6c4984e P 52f06bd6d6a448aeb07ada8c0d89e2e1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8483d5dffddc4ca080e13001a1707d12 in term 0.
I20260806 06:26:55.939157 32683 raft_consensus.cc:2454] T d4315b4746e84442815c3de0e6c4984e P 067b2ec58c34466f99bd66c5780d8fd3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8483d5dffddc4ca080e13001a1707d12 in term 0.
I20260806 06:26:55.939299 551 leader_election.cc:302] T d4315b4746e84442815c3de0e6c4984e P 8483d5dffddc4ca080e13001a1707d12 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52f06bd6d6a448aeb07ada8c0d89e2e1, 8483d5dffddc4ca080e13001a1707d12; no voters:
I20260806 06:26:55.939407 767 raft_consensus.cc:2781] T d4315b4746e84442815c3de0e6c4984e P 8483d5dffddc4ca080e13001a1707d12 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 06:26:55.939460 767 raft_consensus.cc:482] T d4315b4746e84442815c3de0e6c4984e P 8483d5dffddc4ca080e13001a1707d12 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 06:26:55.939486 767 raft_consensus.cc:3037] T d4315b4746e84442815c3de0e6c4984e P 8483d5dffddc4ca080e13001a1707d12 [term 0 FOLLOWER]: Advancing to term 1
I20260806 06:26:55.940193 767 raft_consensus.cc:504] T d4315b4746e84442815c3de0e6c4984e P 8483d5dffddc4ca080e13001a1707d12 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "067b2ec58c34466f99bd66c5780d8fd3" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 44983 } } peers { permanent_uuid: "52f06bd6d6a448aeb07ada8c0d89e2e1" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 39591 } } peers { permanent_uuid: "8483d5dffddc4ca080e13001a1707d12" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 41903 } }
I20260806 06:26:55.940316 767 leader_election.cc:288] T d4315b4746e84442815c3de0e6c4984e P 8483d5dffddc4ca080e13001a1707d12 [CANDIDATE]: Term 1 election: Requested vote from peers 067b2ec58c34466f99bd66c5780d8fd3 (127.21.12.129:44983), 52f06bd6d6a448aeb07ada8c0d89e2e1 (127.21.12.130:39591)
I20260806 06:26:55.940479 32683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4315b4746e84442815c3de0e6c4984e" candidate_uuid: "8483d5dffddc4ca080e13001a1707d12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "067b2ec58c34466f99bd66c5780d8fd3"
I20260806 06:26:55.940470 436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4315b4746e84442815c3de0e6c4984e" candidate_uuid: "8483d5dffddc4ca080e13001a1707d12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52f06bd6d6a448aeb07ada8c0d89e2e1"
I20260806 06:26:55.940559 32683 raft_consensus.cc:3037] T d4315b4746e84442815c3de0e6c4984e P 067b2ec58c34466f99bd66c5780d8fd3 [term 0 FOLLOWER]: Advancing to term 1
I20260806 06:26:55.940579 436 raft_consensus.cc:3037] T d4315b4746e84442815c3de0e6c4984e P 52f06bd6d6a448aeb07ada8c0d89e2e1 [term 0 FOLLOWER]: Advancing to term 1
I20260806 06:26:55.941293 32683 raft_consensus.cc:2454] T d4315b4746e84442815c3de0e6c4984e P 067b2ec58c34466f99bd66c5780d8fd3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8483d5dffddc4ca080e13001a1707d12 in term 1.
I20260806 06:26:55.941291 436 raft_consensus.cc:2454] T d4315b4746e84442815c3de0e6c4984e P 52f06bd6d6a448aeb07ada8c0d89e2e1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8483d5dffddc4ca080e13001a1707d12 in term 1.
I20260806 06:26:55.941493 551 leader_election.cc:302] T d4315b4746e84442815c3de0e6c4984e P 8483d5dffddc4ca080e13001a1707d12 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52f06bd6d6a448aeb07ada8c0d89e2e1, 8483d5dffddc4ca080e13001a1707d12; no voters:
I20260806 06:26:55.941623 767 raft_consensus.cc:2781] T d4315b4746e84442815c3de0e6c4984e P 8483d5dffddc4ca080e13001a1707d12 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 06:26:55.941679 767 raft_consensus.cc:686] T d4315b4746e84442815c3de0e6c4984e P 8483d5dffddc4ca080e13001a1707d12 [term 1 LEADER]: Becoming Leader. State: Replica: 8483d5dffddc4ca080e13001a1707d12, State: Running, Role: LEADER
I20260806 06:26:55.941735 767 consensus_queue.cc:237] T d4315b4746e84442815c3de0e6c4984e P 8483d5dffddc4ca080e13001a1707d12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "067b2ec58c34466f99bd66c5780d8fd3" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 44983 } } peers { permanent_uuid: "52f06bd6d6a448aeb07ada8c0d89e2e1" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 39591 } } peers { permanent_uuid: "8483d5dffddc4ca080e13001a1707d12" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 41903 } }
I20260806 06:26:55.942073 801 tablet_replica.cc:439] T d4315b4746e84442815c3de0e6c4984e P 8483d5dffddc4ca080e13001a1707d12: TxnStatusTablet state changed. Reason: New leader 8483d5dffddc4ca080e13001a1707d12. Latest consensus state: current_term: 1 leader_uuid: "8483d5dffddc4ca080e13001a1707d12" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "067b2ec58c34466f99bd66c5780d8fd3" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 44983 } } peers { permanent_uuid: "52f06bd6d6a448aeb07ada8c0d89e2e1" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 39591 } } peers { permanent_uuid: "8483d5dffddc4ca080e13001a1707d12" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 41903 } } }
I20260806 06:26:55.942158 801 tablet_replica.cc:442] T d4315b4746e84442815c3de0e6c4984e P 8483d5dffddc4ca080e13001a1707d12: This TxnStatusTablet replica's current role is: LEADER
I20260806 06:26:55.942339 820 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260806 06:26:55.942466 820 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260806 06:26:55.942443 32424 catalog_manager.cc:5495] T d4315b4746e84442815c3de0e6c4984e P 8483d5dffddc4ca080e13001a1707d12 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8483d5dffddc4ca080e13001a1707d12 (127.21.12.131). New cstate: current_term: 1 leader_uuid: "8483d5dffddc4ca080e13001a1707d12" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "067b2ec58c34466f99bd66c5780d8fd3" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 44983 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52f06bd6d6a448aeb07ada8c0d89e2e1" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 39591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8483d5dffddc4ca080e13001a1707d12" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 41903 } health_report { overall_health: HEALTHY } } }
I20260806 06:26:55.954916 757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:55.952583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44782 (local address 127.21.12.131:41903)
0806 06:26:55.952629 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0806 06:26:55.952631 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:26:55.952640 (+ 9us) server_negotiation.cc:407] Connection header received
0806 06:26:55.952726 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:26:55.952727 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:26:55.952746 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:26:55.952802 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 06:26:55.952926 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:55.953425 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:55.954153 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:55.954380 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:55.954451 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:55.954618 (+ 167us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 06:26:55.954717 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 06:26:55.954777 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0806 06:26:55.954826 (+ 49us) server_negotiation.cc:299] Negotiation successful
0806 06:26:55.954848 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260806 06:26:55.954936 812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:55.952510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.131:41903 (local address 127.0.0.1:44782)
0806 06:26:55.952595 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0806 06:26:55.952608 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 06:26:55.952659 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:26:55.952815 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:26:55.952816 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:26:55.952817 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 06:26:55.952867 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:55.952869 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:55.953504 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:55.953505 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:55.954080 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:55.954082 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:55.954441 (+ 359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:55.954442 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:55.954510 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:55.954527 (+ 17us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 06:26:55.954772 (+ 245us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 06:26:55.954780 (+ 8us) client_negotiation.cc:769] Sending connection context
0806 06:26:55.954823 (+ 43us) client_negotiation.cc:240] Negotiation successful
0806 06:26:55.954872 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260806 06:26:55.955914 436 raft_consensus.cc:1261] T d4315b4746e84442815c3de0e6c4984e P 52f06bd6d6a448aeb07ada8c0d89e2e1 [term 1 FOLLOWER]: Refusing update from remote peer 8483d5dffddc4ca080e13001a1707d12: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 06:26:55.956007 32683 raft_consensus.cc:1261] T d4315b4746e84442815c3de0e6c4984e P 067b2ec58c34466f99bd66c5780d8fd3 [term 1 FOLLOWER]: Refusing update from remote peer 8483d5dffddc4ca080e13001a1707d12: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 06:26:55.956161 767 consensus_queue.cc:1035] T d4315b4746e84442815c3de0e6c4984e P 8483d5dffddc4ca080e13001a1707d12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52f06bd6d6a448aeb07ada8c0d89e2e1" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 06:26:55.956278 801 consensus_queue.cc:1035] T d4315b4746e84442815c3de0e6c4984e P 8483d5dffddc4ca080e13001a1707d12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "067b2ec58c34466f99bd66c5780d8fd3" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 44983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 06:26:55.956883 769 tablet_replica.cc:439] T d4315b4746e84442815c3de0e6c4984e P 52f06bd6d6a448aeb07ada8c0d89e2e1: TxnStatusTablet state changed. Reason: New leader 8483d5dffddc4ca080e13001a1707d12. Latest consensus state: current_term: 1 leader_uuid: "8483d5dffddc4ca080e13001a1707d12" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "067b2ec58c34466f99bd66c5780d8fd3" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 44983 } } peers { permanent_uuid: "52f06bd6d6a448aeb07ada8c0d89e2e1" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 39591 } } peers { permanent_uuid: "8483d5dffddc4ca080e13001a1707d12" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 41903 } } }
I20260806 06:26:55.956971 769 tablet_replica.cc:442] T d4315b4746e84442815c3de0e6c4984e P 52f06bd6d6a448aeb07ada8c0d89e2e1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 06:26:55.956954 768 tablet_replica.cc:439] T d4315b4746e84442815c3de0e6c4984e P 067b2ec58c34466f99bd66c5780d8fd3: TxnStatusTablet state changed. Reason: New leader 8483d5dffddc4ca080e13001a1707d12. Latest consensus state: current_term: 1 leader_uuid: "8483d5dffddc4ca080e13001a1707d12" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "067b2ec58c34466f99bd66c5780d8fd3" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 44983 } } peers { permanent_uuid: "52f06bd6d6a448aeb07ada8c0d89e2e1" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 39591 } } peers { permanent_uuid: "8483d5dffddc4ca080e13001a1707d12" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 41903 } } }
I20260806 06:26:55.957032 768 tablet_replica.cc:442] T d4315b4746e84442815c3de0e6c4984e P 067b2ec58c34466f99bd66c5780d8fd3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 06:26:55.957607 767 tablet_replica.cc:439] T d4315b4746e84442815c3de0e6c4984e P 8483d5dffddc4ca080e13001a1707d12: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8483d5dffddc4ca080e13001a1707d12" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "067b2ec58c34466f99bd66c5780d8fd3" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 44983 } } peers { permanent_uuid: "52f06bd6d6a448aeb07ada8c0d89e2e1" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 39591 } } peers { permanent_uuid: "8483d5dffddc4ca080e13001a1707d12" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 41903 } } }
I20260806 06:26:55.957680 767 tablet_replica.cc:442] T d4315b4746e84442815c3de0e6c4984e P 8483d5dffddc4ca080e13001a1707d12: This TxnStatusTablet replica's current role is: LEADER
I20260806 06:26:55.957719 767 tablet_replica.cc:439] T d4315b4746e84442815c3de0e6c4984e P 8483d5dffddc4ca080e13001a1707d12: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8483d5dffddc4ca080e13001a1707d12" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "067b2ec58c34466f99bd66c5780d8fd3" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 44983 } } peers { permanent_uuid: "52f06bd6d6a448aeb07ada8c0d89e2e1" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 39591 } } peers { permanent_uuid: "8483d5dffddc4ca080e13001a1707d12" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 41903 } } }
I20260806 06:26:55.957752 767 tablet_replica.cc:442] T d4315b4746e84442815c3de0e6c4984e P 8483d5dffddc4ca080e13001a1707d12: This TxnStatusTablet replica's current role is: LEADER
I20260806 06:26:55.957897 769 tablet_replica.cc:439] T d4315b4746e84442815c3de0e6c4984e P 52f06bd6d6a448aeb07ada8c0d89e2e1: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8483d5dffddc4ca080e13001a1707d12" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "067b2ec58c34466f99bd66c5780d8fd3" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 44983 } } peers { permanent_uuid: "52f06bd6d6a448aeb07ada8c0d89e2e1" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 39591 } } peers { permanent_uuid: "8483d5dffddc4ca080e13001a1707d12" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 41903 } } }
I20260806 06:26:55.957973 769 tablet_replica.cc:442] T d4315b4746e84442815c3de0e6c4984e P 52f06bd6d6a448aeb07ada8c0d89e2e1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 06:26:55.957979 779 mvcc.cc:205] Tried to move back new op lower bound from 7315446234953777152 to 7315446234897920000. Current Snapshot: MvccSnapshot[applied={T|T < 7315446234953777152}]
I20260806 06:26:55.958168 768 tablet_replica.cc:439] T d4315b4746e84442815c3de0e6c4984e P 067b2ec58c34466f99bd66c5780d8fd3: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8483d5dffddc4ca080e13001a1707d12" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "067b2ec58c34466f99bd66c5780d8fd3" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 44983 } } peers { permanent_uuid: "52f06bd6d6a448aeb07ada8c0d89e2e1" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 39591 } } peers { permanent_uuid: "8483d5dffddc4ca080e13001a1707d12" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 41903 } } }
I20260806 06:26:55.958233 768 tablet_replica.cc:442] T d4315b4746e84442815c3de0e6c4984e P 067b2ec58c34466f99bd66c5780d8fd3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 06:26:55.961398 757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:55.959458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44794 (local address 127.21.12.131:41903)
0806 06:26:55.959508 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0806 06:26:55.959510 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:26:55.959516 (+ 6us) server_negotiation.cc:407] Connection header received
0806 06:26:55.959543 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:26:55.959543 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:26:55.959551 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:26:55.959593 (+ 42us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 06:26:55.959712 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:55.960146 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:55.960829 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:55.961016 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:55.961068 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:55.961194 (+ 126us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 06:26:55.961224 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 06:26:55.961265 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0806 06:26:55.961323 (+ 58us) server_negotiation.cc:299] Negotiation successful
0806 06:26:55.961338 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260806 06:26:55.966022 757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:55.963572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.131:43177 (local address 127.21.12.131:41903)
0806 06:26:55.963623 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0806 06:26:55.963625 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:26:55.963726 (+ 101us) server_negotiation.cc:407] Connection header received
0806 06:26:55.963785 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:26:55.963786 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:26:55.963795 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:26:55.963848 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 06:26:55.963934 (+ 86us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:55.964623 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:55.965324 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:55.965621 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:55.965687 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:55.965801 (+ 114us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 06:26:55.965830 (+ 29us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 06:26:55.965876 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0806 06:26:55.965919 (+ 43us) server_negotiation.cc:299] Negotiation successful
0806 06:26:55.965935 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260806 06:26:55.966044 832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:55.963516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.131:41903 (local address 127.21.12.131:43177)
0806 06:26:55.963670 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0806 06:26:55.963679 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0806 06:26:55.963732 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:26:55.963850 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:26:55.963850 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:26:55.963857 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 06:26:55.963900 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:55.963901 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:55.964689 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:55.964690 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:55.965259 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:55.965260 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:55.965684 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:55.965685 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:55.965728 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:55.965740 (+ 12us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 06:26:55.965885 (+ 145us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 06:26:55.965886 (+ 1us) client_negotiation.cc:769] Sending connection context
0806 06:26:55.965915 (+ 29us) client_negotiation.cc:240] Negotiation successful
0806 06:26:55.965943 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":82,"threads_started":1}
I20260806 06:26:55.968281 32683 raft_consensus.cc:1261] T e698bde980464463a3b755b465be490d P 067b2ec58c34466f99bd66c5780d8fd3 [term 1 FOLLOWER]: Refusing update from remote peer 8483d5dffddc4ca080e13001a1707d12: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 06:26:55.968286 436 raft_consensus.cc:1261] T e698bde980464463a3b755b465be490d P 52f06bd6d6a448aeb07ada8c0d89e2e1 [term 1 FOLLOWER]: Refusing update from remote peer 8483d5dffddc4ca080e13001a1707d12: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 06:26:55.968492 767 consensus_queue.cc:1035] T e698bde980464463a3b755b465be490d P 8483d5dffddc4ca080e13001a1707d12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "067b2ec58c34466f99bd66c5780d8fd3" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 44983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 06:26:55.968634 801 consensus_queue.cc:1035] T e698bde980464463a3b755b465be490d P 8483d5dffddc4ca080e13001a1707d12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52f06bd6d6a448aeb07ada8c0d89e2e1" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 06:26:55.982419 32425 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33140:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 06:26:55.982527 32425 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33140:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 06:26:55.983155 32425 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 987ca9a955c54e639f87808644890c4c: Sending DeleteTablet for 3 replicas of tablet e698bde980464463a3b755b465be490d
I20260806 06:26:55.983441 614 tablet_service.cc:1514] Processing DeleteTablet for tablet e698bde980464463a3b755b465be490d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 06:26:55 UTC) from {username='oryx', principal='oryx/127.21.12.190@KRBTEST.COM'} at 127.0.0.1:44776
I20260806 06:26:55.983421 32651 tablet_service.cc:1514] Processing DeleteTablet for tablet e698bde980464463a3b755b465be490d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 06:26:55 UTC) from {username='oryx', principal='oryx/127.21.12.190@KRBTEST.COM'} at 127.0.0.1:55198
I20260806 06:26:55.983546 791 tablet_replica.cc:330] T e698bde980464463a3b755b465be490d P 067b2ec58c34466f99bd66c5780d8fd3: stopping tablet replica
I20260806 06:26:55.983546 792 tablet_replica.cc:330] T e698bde980464463a3b755b465be490d P 8483d5dffddc4ca080e13001a1707d12: stopping tablet replica
I20260806 06:26:55.983604 791 raft_consensus.cc:2229] T e698bde980464463a3b755b465be490d P 067b2ec58c34466f99bd66c5780d8fd3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 06:26:55.983618 792 raft_consensus.cc:2229] T e698bde980464463a3b755b465be490d P 8483d5dffddc4ca080e13001a1707d12 [term 1 LEADER]: Raft consensus shutting down.
I20260806 06:26:55.983639 791 raft_consensus.cc:2258] T e698bde980464463a3b755b465be490d P 067b2ec58c34466f99bd66c5780d8fd3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 06:26:55.983700 792 raft_consensus.cc:2258] T e698bde980464463a3b755b465be490d P 8483d5dffddc4ca080e13001a1707d12 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 06:26:55.983914 404 tablet_service.cc:1514] Processing DeleteTablet for tablet e698bde980464463a3b755b465be490d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 06:26:55 UTC) from {username='oryx', principal='oryx/127.21.12.190@KRBTEST.COM'} at 127.0.0.1:39336
I20260806 06:26:55.984005 795 tablet_replica.cc:330] T e698bde980464463a3b755b465be490d P 52f06bd6d6a448aeb07ada8c0d89e2e1: stopping tablet replica
I20260806 06:26:55.984050 795 raft_consensus.cc:2229] T e698bde980464463a3b755b465be490d P 52f06bd6d6a448aeb07ada8c0d89e2e1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 06:26:55.984090 795 raft_consensus.cc:2258] T e698bde980464463a3b755b465be490d P 52f06bd6d6a448aeb07ada8c0d89e2e1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 06:26:55.984139 792 ts_tablet_manager.cc:1905] T e698bde980464463a3b755b465be490d P 8483d5dffddc4ca080e13001a1707d12: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 06:26:55.984740 795 ts_tablet_manager.cc:1905] T e698bde980464463a3b755b465be490d P 52f06bd6d6a448aeb07ada8c0d89e2e1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 06:26:55.985759 792 ts_tablet_manager.cc:1918] T e698bde980464463a3b755b465be490d P 8483d5dffddc4ca080e13001a1707d12: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 06:26:55.985807 792 log.cc:1196] T e698bde980464463a3b755b465be490d P 8483d5dffddc4ca080e13001a1707d12: Deleting WAL directory at /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785997581049705-21554-0/minicluster-data/ts-2/wal/wals/e698bde980464463a3b755b465be490d
I20260806 06:26:55.986029 792 ts_tablet_manager.cc:1939] T e698bde980464463a3b755b465be490d P 8483d5dffddc4ca080e13001a1707d12: Deleting consensus metadata
I20260806 06:26:55.986361 795 ts_tablet_manager.cc:1918] T e698bde980464463a3b755b465be490d P 52f06bd6d6a448aeb07ada8c0d89e2e1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 06:26:55.986418 795 log.cc:1196] T e698bde980464463a3b755b465be490d P 52f06bd6d6a448aeb07ada8c0d89e2e1: Deleting WAL directory at /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785997581049705-21554-0/minicluster-data/ts-1/wal/wals/e698bde980464463a3b755b465be490d
I20260806 06:26:55.986620 795 ts_tablet_manager.cc:1939] T e698bde980464463a3b755b465be490d P 52f06bd6d6a448aeb07ada8c0d89e2e1: Deleting consensus metadata
I20260806 06:26:55.987076 32406 catalog_manager.cc:4841] TS 52f06bd6d6a448aeb07ada8c0d89e2e1 (127.21.12.130:39591): tablet e698bde980464463a3b755b465be490d (table test-table [id=27460a7733f84b83873d847f33ac3698]) successfully deleted
I20260806 06:26:55.987408 791 ts_tablet_manager.cc:1905] T e698bde980464463a3b755b465be490d P 067b2ec58c34466f99bd66c5780d8fd3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 06:26:55.988967 791 ts_tablet_manager.cc:1918] T e698bde980464463a3b755b465be490d P 067b2ec58c34466f99bd66c5780d8fd3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 06:26:55.989019 791 log.cc:1196] T e698bde980464463a3b755b465be490d P 067b2ec58c34466f99bd66c5780d8fd3: Deleting WAL directory at /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785997581049705-21554-0/minicluster-data/ts-0/wal/wals/e698bde980464463a3b755b465be490d
I20260806 06:26:55.989218 791 ts_tablet_manager.cc:1939] T e698bde980464463a3b755b465be490d P 067b2ec58c34466f99bd66c5780d8fd3: Deleting consensus metadata
I20260806 06:26:55.989547 32407 catalog_manager.cc:4841] TS 067b2ec58c34466f99bd66c5780d8fd3 (127.21.12.129:44983): tablet e698bde980464463a3b755b465be490d (table test-table [id=27460a7733f84b83873d847f33ac3698]) successfully deleted
I20260806 06:26:55.989694 32406 catalog_manager.cc:4841] TS 8483d5dffddc4ca080e13001a1707d12 (127.21.12.131:41903): tablet e698bde980464463a3b755b465be490d (table test-table [id=27460a7733f84b83873d847f33ac3698]) successfully deleted
Aug 06 06:26:55 dist-test-slave-4tdj krb5kdc[32356](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997615, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 06 06:26:56 dist-test-slave-4tdj krb5kdc[32356](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.21.12.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 06:26:56 dist-test-slave-4tdj krb5kdc[32356](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.21.12.190@KRBTEST.COM, Server not found in Kerberos database
W20260806 06:26:56.009228 32549 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 06:26:56.003028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33162 (local address 127.21.12.190:33805)
0806 06:26:56.003101 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0806 06:26:56.003103 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:26:56.003204 (+ 101us) server_negotiation.cc:407] Connection header received
0806 06:26:56.003278 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:26:56.003278 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:26:56.003289 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:26:56.003343 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:26:56.004045 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:56.004551 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:56.005569 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:56.005763 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:56.005823 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:56.009160 (+ 3337us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33162: BlockingRecv error: recv got EOF from 127.0.0.1:33162 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Aug 06 06:26:56 dist-test-slave-4tdj krb5kdc[32356](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997615, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.21.12.190@KRBTEST.COM
I20260806 06:26:56.017786 32549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:56.010253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33178 (local address 127.21.12.190:33805)
0806 06:26:56.010318 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0806 06:26:56.010320 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:26:56.010363 (+ 43us) server_negotiation.cc:407] Connection header received
0806 06:26:56.010443 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:26:56.010444 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:26:56.010454 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:26:56.010501 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:26:56.010926 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:56.011658 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:56.012312 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:56.012614 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:56.012683 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:56.014977 (+ 2294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:26:56.014979 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:26:56.014980 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 06:26:56.014987 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:26:56.016699 (+ 1712us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:26:56.017039 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:26:56.017040 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:26:56.017040 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:26:56.017104 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:26:56.017283 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:26:56.017284 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:26:56.017285 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:26:56.017404 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:26:56.017503 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0806 06:26:56.017611 (+ 108us) server_negotiation.cc:299] Negotiation successful
0806 06:26:56.017689 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260806 06:26:56.019421 32425 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33178:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 06:26:56.019591 32425 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 06:26:56.021086 32651 tablet_service.cc:1467] Processing CreateTablet for tablet d201ebe1653f492e94624c09887335fc (DEFAULT_TABLE table=test-table [id=e65089315d3e4c66af237255012c0da6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 06:26:56.021152 404 tablet_service.cc:1467] Processing CreateTablet for tablet d201ebe1653f492e94624c09887335fc (DEFAULT_TABLE table=test-table [id=e65089315d3e4c66af237255012c0da6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 06:26:56.021252 32651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d201ebe1653f492e94624c09887335fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 06:26:56.021267 404 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d201ebe1653f492e94624c09887335fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 06:26:56.022454 614 tablet_service.cc:1467] Processing CreateTablet for tablet d201ebe1653f492e94624c09887335fc (DEFAULT_TABLE table=test-table [id=e65089315d3e4c66af237255012c0da6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 06:26:56.022607 614 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d201ebe1653f492e94624c09887335fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 06:26:56.022655 761 tablet_bootstrap.cc:492] T d201ebe1653f492e94624c09887335fc P 52f06bd6d6a448aeb07ada8c0d89e2e1: Bootstrap starting.
I20260806 06:26:56.022658 762 tablet_bootstrap.cc:492] T d201ebe1653f492e94624c09887335fc P 067b2ec58c34466f99bd66c5780d8fd3: Bootstrap starting.
I20260806 06:26:56.022976 762 tablet_bootstrap.cc:654] T d201ebe1653f492e94624c09887335fc P 067b2ec58c34466f99bd66c5780d8fd3: Neither blocks nor log segments found. Creating new log.
I20260806 06:26:56.023069 761 tablet_bootstrap.cc:654] T d201ebe1653f492e94624c09887335fc P 52f06bd6d6a448aeb07ada8c0d89e2e1: Neither blocks nor log segments found. Creating new log.
I20260806 06:26:56.023691 762 tablet_bootstrap.cc:492] T d201ebe1653f492e94624c09887335fc P 067b2ec58c34466f99bd66c5780d8fd3: No bootstrap required, opened a new log
I20260806 06:26:56.023741 762 ts_tablet_manager.cc:1397] T d201ebe1653f492e94624c09887335fc P 067b2ec58c34466f99bd66c5780d8fd3: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260806 06:26:56.023762 761 tablet_bootstrap.cc:492] T d201ebe1653f492e94624c09887335fc P 52f06bd6d6a448aeb07ada8c0d89e2e1: No bootstrap required, opened a new log
I20260806 06:26:56.023803 761 ts_tablet_manager.cc:1397] T d201ebe1653f492e94624c09887335fc P 52f06bd6d6a448aeb07ada8c0d89e2e1: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260806 06:26:56.023856 762 raft_consensus.cc:348] T d201ebe1653f492e94624c09887335fc P 067b2ec58c34466f99bd66c5780d8fd3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "067b2ec58c34466f99bd66c5780d8fd3" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 44983 } } peers { permanent_uuid: "52f06bd6d6a448aeb07ada8c0d89e2e1" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 39591 } } peers { permanent_uuid: "8483d5dffddc4ca080e13001a1707d12" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 41903 } }
I20260806 06:26:56.023926 762 raft_consensus.cc:374] T d201ebe1653f492e94624c09887335fc P 067b2ec58c34466f99bd66c5780d8fd3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 06:26:56.023947 762 raft_consensus.cc:729] T d201ebe1653f492e94624c09887335fc P 067b2ec58c34466f99bd66c5780d8fd3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 067b2ec58c34466f99bd66c5780d8fd3, State: Initialized, Role: FOLLOWER
I20260806 06:26:56.023943 761 raft_consensus.cc:348] T d201ebe1653f492e94624c09887335fc P 52f06bd6d6a448aeb07ada8c0d89e2e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "067b2ec58c34466f99bd66c5780d8fd3" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 44983 } } peers { permanent_uuid: "52f06bd6d6a448aeb07ada8c0d89e2e1" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 39591 } } peers { permanent_uuid: "8483d5dffddc4ca080e13001a1707d12" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 41903 } }
I20260806 06:26:56.023994 762 consensus_queue.cc:260] T d201ebe1653f492e94624c09887335fc P 067b2ec58c34466f99bd66c5780d8fd3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "067b2ec58c34466f99bd66c5780d8fd3" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 44983 } } peers { permanent_uuid: "52f06bd6d6a448aeb07ada8c0d89e2e1" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 39591 } } peers { permanent_uuid: "8483d5dffddc4ca080e13001a1707d12" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 41903 } }
I20260806 06:26:56.024022 761 raft_consensus.cc:374] T d201ebe1653f492e94624c09887335fc P 52f06bd6d6a448aeb07ada8c0d89e2e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 06:26:56.024045 761 raft_consensus.cc:729] T d201ebe1653f492e94624c09887335fc P 52f06bd6d6a448aeb07ada8c0d89e2e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52f06bd6d6a448aeb07ada8c0d89e2e1, State: Initialized, Role: FOLLOWER
I20260806 06:26:56.024088 762 ts_tablet_manager.cc:1428] T d201ebe1653f492e94624c09887335fc P 067b2ec58c34466f99bd66c5780d8fd3: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260806 06:26:56.024086 761 consensus_queue.cc:260] T d201ebe1653f492e94624c09887335fc P 52f06bd6d6a448aeb07ada8c0d89e2e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "067b2ec58c34466f99bd66c5780d8fd3" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 44983 } } peers { permanent_uuid: "52f06bd6d6a448aeb07ada8c0d89e2e1" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 39591 } } peers { permanent_uuid: "8483d5dffddc4ca080e13001a1707d12" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 41903 } }
I20260806 06:26:56.024171 761 ts_tablet_manager.cc:1428] T d201ebe1653f492e94624c09887335fc P 52f06bd6d6a448aeb07ada8c0d89e2e1: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260806 06:26:56.024194 759 tablet_bootstrap.cc:492] T d201ebe1653f492e94624c09887335fc P 8483d5dffddc4ca080e13001a1707d12: Bootstrap starting.
I20260806 06:26:56.024591 759 tablet_bootstrap.cc:654] T d201ebe1653f492e94624c09887335fc P 8483d5dffddc4ca080e13001a1707d12: Neither blocks nor log segments found. Creating new log.
I20260806 06:26:56.025281 759 tablet_bootstrap.cc:492] T d201ebe1653f492e94624c09887335fc P 8483d5dffddc4ca080e13001a1707d12: No bootstrap required, opened a new log
I20260806 06:26:56.025336 759 ts_tablet_manager.cc:1397] T d201ebe1653f492e94624c09887335fc P 8483d5dffddc4ca080e13001a1707d12: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260806 06:26:56.025472 759 raft_consensus.cc:348] T d201ebe1653f492e94624c09887335fc P 8483d5dffddc4ca080e13001a1707d12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "067b2ec58c34466f99bd66c5780d8fd3" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 44983 } } peers { permanent_uuid: "52f06bd6d6a448aeb07ada8c0d89e2e1" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 39591 } } peers { permanent_uuid: "8483d5dffddc4ca080e13001a1707d12" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 41903 } }
I20260806 06:26:56.025535 759 raft_consensus.cc:374] T d201ebe1653f492e94624c09887335fc P 8483d5dffddc4ca080e13001a1707d12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 06:26:56.025553 759 raft_consensus.cc:729] T d201ebe1653f492e94624c09887335fc P 8483d5dffddc4ca080e13001a1707d12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8483d5dffddc4ca080e13001a1707d12, State: Initialized, Role: FOLLOWER
I20260806 06:26:56.025611 759 consensus_queue.cc:260] T d201ebe1653f492e94624c09887335fc P 8483d5dffddc4ca080e13001a1707d12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "067b2ec58c34466f99bd66c5780d8fd3" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 44983 } } peers { permanent_uuid: "52f06bd6d6a448aeb07ada8c0d89e2e1" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 39591 } } peers { permanent_uuid: "8483d5dffddc4ca080e13001a1707d12" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 41903 } }
I20260806 06:26:56.025702 759 ts_tablet_manager.cc:1428] T d201ebe1653f492e94624c09887335fc P 8483d5dffddc4ca080e13001a1707d12: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260806 06:26:56.072604 769 raft_consensus.cc:482] T d201ebe1653f492e94624c09887335fc P 52f06bd6d6a448aeb07ada8c0d89e2e1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 06:26:56.072690 769 raft_consensus.cc:504] T d201ebe1653f492e94624c09887335fc P 52f06bd6d6a448aeb07ada8c0d89e2e1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "067b2ec58c34466f99bd66c5780d8fd3" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 44983 } } peers { permanent_uuid: "52f06bd6d6a448aeb07ada8c0d89e2e1" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 39591 } } peers { permanent_uuid: "8483d5dffddc4ca080e13001a1707d12" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 41903 } }
I20260806 06:26:56.073002 769 leader_election.cc:288] T d201ebe1653f492e94624c09887335fc P 52f06bd6d6a448aeb07ada8c0d89e2e1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 067b2ec58c34466f99bd66c5780d8fd3 (127.21.12.129:44983), 8483d5dffddc4ca080e13001a1707d12 (127.21.12.131:41903)
I20260806 06:26:56.076865 755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:56.073225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.130:43767 (local address 127.21.12.129:44983)
0806 06:26:56.073274 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0806 06:26:56.073276 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:26:56.073358 (+ 82us) server_negotiation.cc:407] Connection header received
0806 06:26:56.073452 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:26:56.073453 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:26:56.073463 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:26:56.073492 (+ 29us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 06:26:56.073732 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:56.074198 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:56.075998 (+ 1800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:56.076523 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:56.076611 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:56.076622 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0806 06:26:56.076755 (+ 133us) server_negotiation.cc:299] Negotiation successful
0806 06:26:56.076803 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260806 06:26:56.076874 861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:56.073160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.129:44983 (local address 127.21.12.130:43767)
0806 06:26:56.073294 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0806 06:26:56.073305 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0806 06:26:56.073379 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:26:56.073563 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:26:56.073564 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:26:56.073565 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 06:26:56.073648 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:56.073649 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:56.074275 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:56.074276 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:56.075900 (+ 1624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:56.075903 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:56.076625 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:56.076625 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:56.076683 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:56.076686 (+ 3us) client_negotiation.cc:769] Sending connection context
0806 06:26:56.076734 (+ 48us) client_negotiation.cc:240] Negotiation successful
0806 06:26:56.076777 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"mutex_wait_us":53,"thread_start_us":56,"threads_started":1}
I20260806 06:26:56.077250 32683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d201ebe1653f492e94624c09887335fc" candidate_uuid: "52f06bd6d6a448aeb07ada8c0d89e2e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "067b2ec58c34466f99bd66c5780d8fd3" is_pre_election: true
I20260806 06:26:56.077349 32683 raft_consensus.cc:2454] T d201ebe1653f492e94624c09887335fc P 067b2ec58c34466f99bd66c5780d8fd3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52f06bd6d6a448aeb07ada8c0d89e2e1 in term 0.
I20260806 06:26:56.077440 857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:56.073168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.131:41903 (local address 127.21.12.130:39641)
0806 06:26:56.073311 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0806 06:26:56.073321 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0806 06:26:56.073368 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:26:56.073565 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:26:56.073565 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:26:56.073566 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 06:26:56.073638 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:56.073641 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:56.074560 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:56.074561 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:56.075904 (+ 1343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:56.075906 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:56.077203 (+ 1297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:56.077204 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:56.077272 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:56.077276 (+ 4us) client_negotiation.cc:769] Sending connection context
0806 06:26:56.077319 (+ 43us) client_negotiation.cc:240] Negotiation successful
0806 06:26:56.077358 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"mutex_wait_us":75,"thread_start_us":53,"threads_started":1}
I20260806 06:26:56.077445 757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:56.073312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.130:39641 (local address 127.21.12.131:41903)
0806 06:26:56.073353 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0806 06:26:56.073355 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:26:56.073364 (+ 9us) server_negotiation.cc:407] Connection header received
0806 06:26:56.073452 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:26:56.073454 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:26:56.073463 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:26:56.073492 (+ 29us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 06:26:56.073709 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:56.074454 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:56.076656 (+ 2202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:56.077127 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:56.077185 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:56.077193 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0806 06:26:56.077340 (+ 147us) server_negotiation.cc:299] Negotiation successful
0806 06:26:56.077386 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260806 06:26:56.077533 333 leader_election.cc:302] T d201ebe1653f492e94624c09887335fc P 52f06bd6d6a448aeb07ada8c0d89e2e1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 067b2ec58c34466f99bd66c5780d8fd3, 52f06bd6d6a448aeb07ada8c0d89e2e1; no voters:
I20260806 06:26:56.077674 769 raft_consensus.cc:2781] T d201ebe1653f492e94624c09887335fc P 52f06bd6d6a448aeb07ada8c0d89e2e1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 06:26:56.077701 769 raft_consensus.cc:482] T d201ebe1653f492e94624c09887335fc P 52f06bd6d6a448aeb07ada8c0d89e2e1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 06:26:56.077715 769 raft_consensus.cc:3037] T d201ebe1653f492e94624c09887335fc P 52f06bd6d6a448aeb07ada8c0d89e2e1 [term 0 FOLLOWER]: Advancing to term 1
I20260806 06:26:56.077849 648 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d201ebe1653f492e94624c09887335fc" candidate_uuid: "52f06bd6d6a448aeb07ada8c0d89e2e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8483d5dffddc4ca080e13001a1707d12" is_pre_election: true
I20260806 06:26:56.077924 648 raft_consensus.cc:2454] T d201ebe1653f492e94624c09887335fc P 8483d5dffddc4ca080e13001a1707d12 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52f06bd6d6a448aeb07ada8c0d89e2e1 in term 0.
I20260806 06:26:56.078579 769 raft_consensus.cc:504] T d201ebe1653f492e94624c09887335fc P 52f06bd6d6a448aeb07ada8c0d89e2e1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "067b2ec58c34466f99bd66c5780d8fd3" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 44983 } } peers { permanent_uuid: "52f06bd6d6a448aeb07ada8c0d89e2e1" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 39591 } } peers { permanent_uuid: "8483d5dffddc4ca080e13001a1707d12" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 41903 } }
I20260806 06:26:56.078696 769 leader_election.cc:288] T d201ebe1653f492e94624c09887335fc P 52f06bd6d6a448aeb07ada8c0d89e2e1 [CANDIDATE]: Term 1 election: Requested vote from peers 067b2ec58c34466f99bd66c5780d8fd3 (127.21.12.129:44983), 8483d5dffddc4ca080e13001a1707d12 (127.21.12.131:41903)
I20260806 06:26:56.078846 32683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d201ebe1653f492e94624c09887335fc" candidate_uuid: "52f06bd6d6a448aeb07ada8c0d89e2e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "067b2ec58c34466f99bd66c5780d8fd3"
I20260806 06:26:56.078925 32683 raft_consensus.cc:3037] T d201ebe1653f492e94624c09887335fc P 067b2ec58c34466f99bd66c5780d8fd3 [term 0 FOLLOWER]: Advancing to term 1
I20260806 06:26:56.078893 648 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d201ebe1653f492e94624c09887335fc" candidate_uuid: "52f06bd6d6a448aeb07ada8c0d89e2e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8483d5dffddc4ca080e13001a1707d12"
I20260806 06:26:56.078969 648 raft_consensus.cc:3037] T d201ebe1653f492e94624c09887335fc P 8483d5dffddc4ca080e13001a1707d12 [term 0 FOLLOWER]: Advancing to term 1
I20260806 06:26:56.079551 648 raft_consensus.cc:2454] T d201ebe1653f492e94624c09887335fc P 8483d5dffddc4ca080e13001a1707d12 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52f06bd6d6a448aeb07ada8c0d89e2e1 in term 1.
I20260806 06:26:56.079720 332 leader_election.cc:302] T d201ebe1653f492e94624c09887335fc P 52f06bd6d6a448aeb07ada8c0d89e2e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52f06bd6d6a448aeb07ada8c0d89e2e1, 8483d5dffddc4ca080e13001a1707d12; no voters:
I20260806 06:26:56.079717 32683 raft_consensus.cc:2454] T d201ebe1653f492e94624c09887335fc P 067b2ec58c34466f99bd66c5780d8fd3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52f06bd6d6a448aeb07ada8c0d89e2e1 in term 1.
I20260806 06:26:56.079798 769 raft_consensus.cc:2781] T d201ebe1653f492e94624c09887335fc P 52f06bd6d6a448aeb07ada8c0d89e2e1 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 06:26:56.079919 769 raft_consensus.cc:686] T d201ebe1653f492e94624c09887335fc P 52f06bd6d6a448aeb07ada8c0d89e2e1 [term 1 LEADER]: Becoming Leader. State: Replica: 52f06bd6d6a448aeb07ada8c0d89e2e1, State: Running, Role: LEADER
I20260806 06:26:56.079990 769 consensus_queue.cc:237] T d201ebe1653f492e94624c09887335fc P 52f06bd6d6a448aeb07ada8c0d89e2e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "067b2ec58c34466f99bd66c5780d8fd3" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 44983 } } peers { permanent_uuid: "52f06bd6d6a448aeb07ada8c0d89e2e1" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 39591 } } peers { permanent_uuid: "8483d5dffddc4ca080e13001a1707d12" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 41903 } }
I20260806 06:26:56.080724 32422 catalog_manager.cc:5495] T d201ebe1653f492e94624c09887335fc P 52f06bd6d6a448aeb07ada8c0d89e2e1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 52f06bd6d6a448aeb07ada8c0d89e2e1 (127.21.12.130). New cstate: current_term: 1 leader_uuid: "52f06bd6d6a448aeb07ada8c0d89e2e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "067b2ec58c34466f99bd66c5780d8fd3" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 44983 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52f06bd6d6a448aeb07ada8c0d89e2e1" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 39591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8483d5dffddc4ca080e13001a1707d12" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 41903 } health_report { overall_health: UNKNOWN } } }
I20260806 06:26:56.084975 753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:56.082863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39346 (local address 127.21.12.130:39591)
0806 06:26:56.082910 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0806 06:26:56.082912 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:26:56.082922 (+ 10us) server_negotiation.cc:407] Connection header received
0806 06:26:56.082955 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:26:56.082955 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:26:56.082974 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:26:56.083072 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 06:26:56.083167 (+ 95us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:56.083605 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:56.084297 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:56.084488 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:56.084536 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:56.084685 (+ 149us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 06:26:56.084788 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 06:26:56.084849 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0806 06:26:56.084897 (+ 48us) server_negotiation.cc:299] Negotiation successful
0806 06:26:56.084914 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260806 06:26:56.085845 648 raft_consensus.cc:1261] T d201ebe1653f492e94624c09887335fc P 8483d5dffddc4ca080e13001a1707d12 [term 1 FOLLOWER]: Refusing update from remote peer 52f06bd6d6a448aeb07ada8c0d89e2e1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 06:26:56.085865 32683 raft_consensus.cc:1261] T d201ebe1653f492e94624c09887335fc P 067b2ec58c34466f99bd66c5780d8fd3 [term 1 FOLLOWER]: Refusing update from remote peer 52f06bd6d6a448aeb07ada8c0d89e2e1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 06:26:56.086086 769 consensus_queue.cc:1035] T d201ebe1653f492e94624c09887335fc P 52f06bd6d6a448aeb07ada8c0d89e2e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "067b2ec58c34466f99bd66c5780d8fd3" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 44983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 06:26:56.086200 863 consensus_queue.cc:1035] T d201ebe1653f492e94624c09887335fc P 52f06bd6d6a448aeb07ada8c0d89e2e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8483d5dffddc4ca080e13001a1707d12" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 41903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 06:26:56.089699 32425 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33178:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 06:26:56.089789 32425 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33178:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 06:26:56.090394 32425 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 987ca9a955c54e639f87808644890c4c: Sending DeleteTablet for 3 replicas of tablet d201ebe1653f492e94624c09887335fc
I20260806 06:26:56.090636 32651 tablet_service.cc:1514] Processing DeleteTablet for tablet d201ebe1653f492e94624c09887335fc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 06:26:56 UTC) from {username='oryx', principal='oryx/127.21.12.190@KRBTEST.COM'} at 127.0.0.1:55198
I20260806 06:26:56.090723 791 tablet_replica.cc:330] T d201ebe1653f492e94624c09887335fc P 067b2ec58c34466f99bd66c5780d8fd3: stopping tablet replica
I20260806 06:26:56.090773 791 raft_consensus.cc:2229] T d201ebe1653f492e94624c09887335fc P 067b2ec58c34466f99bd66c5780d8fd3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 06:26:56.090809 791 raft_consensus.cc:2258] T d201ebe1653f492e94624c09887335fc P 067b2ec58c34466f99bd66c5780d8fd3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 06:26:56.090910 404 tablet_service.cc:1514] Processing DeleteTablet for tablet d201ebe1653f492e94624c09887335fc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 06:26:56 UTC) from {username='oryx', principal='oryx/127.21.12.190@KRBTEST.COM'} at 127.0.0.1:39336
I20260806 06:26:56.090951 614 tablet_service.cc:1514] Processing DeleteTablet for tablet d201ebe1653f492e94624c09887335fc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 06:26:56 UTC) from {username='oryx', principal='oryx/127.21.12.190@KRBTEST.COM'} at 127.0.0.1:44776
I20260806 06:26:56.090986 795 tablet_replica.cc:330] T d201ebe1653f492e94624c09887335fc P 52f06bd6d6a448aeb07ada8c0d89e2e1: stopping tablet replica
I20260806 06:26:56.091032 792 tablet_replica.cc:330] T d201ebe1653f492e94624c09887335fc P 8483d5dffddc4ca080e13001a1707d12: stopping tablet replica
I20260806 06:26:56.091041 795 raft_consensus.cc:2229] T d201ebe1653f492e94624c09887335fc P 52f06bd6d6a448aeb07ada8c0d89e2e1 [term 1 LEADER]: Raft consensus shutting down.
I20260806 06:26:56.091076 792 raft_consensus.cc:2229] T d201ebe1653f492e94624c09887335fc P 8483d5dffddc4ca080e13001a1707d12 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 06:26:56.091118 792 raft_consensus.cc:2258] T d201ebe1653f492e94624c09887335fc P 8483d5dffddc4ca080e13001a1707d12 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 06:26:56.091118 795 raft_consensus.cc:2258] T d201ebe1653f492e94624c09887335fc P 52f06bd6d6a448aeb07ada8c0d89e2e1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 06:26:56.091203 32425 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33178:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 06:26:56.091329 32425 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 06:26:56.091971 791 ts_tablet_manager.cc:1905] T d201ebe1653f492e94624c09887335fc P 067b2ec58c34466f99bd66c5780d8fd3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 06:26:56.092052 792 ts_tablet_manager.cc:1905] T d201ebe1653f492e94624c09887335fc P 8483d5dffddc4ca080e13001a1707d12: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 06:26:56.092607 32651 tablet_service.cc:1467] Processing CreateTablet for tablet 62977782ae3b4af1becfad2d4bd056c4 (DEFAULT_TABLE table=test-table [id=a30f3e4e3ed14439af73504568fef3e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 06:26:56.092777 32651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62977782ae3b4af1becfad2d4bd056c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 06:26:56.092981 795 ts_tablet_manager.cc:1905] T d201ebe1653f492e94624c09887335fc P 52f06bd6d6a448aeb07ada8c0d89e2e1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 06:26:56.093964 762 tablet_bootstrap.cc:492] T 62977782ae3b4af1becfad2d4bd056c4 P 067b2ec58c34466f99bd66c5780d8fd3: Bootstrap starting.
I20260806 06:26:56.094252 795 ts_tablet_manager.cc:1918] T d201ebe1653f492e94624c09887335fc P 52f06bd6d6a448aeb07ada8c0d89e2e1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 06:26:56.094310 762 tablet_bootstrap.cc:654] T 62977782ae3b4af1becfad2d4bd056c4 P 067b2ec58c34466f99bd66c5780d8fd3: Neither blocks nor log segments found. Creating new log.
I20260806 06:26:56.094323 795 log.cc:1196] T d201ebe1653f492e94624c09887335fc P 52f06bd6d6a448aeb07ada8c0d89e2e1: Deleting WAL directory at /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785997581049705-21554-0/minicluster-data/ts-1/wal/wals/d201ebe1653f492e94624c09887335fc
I20260806 06:26:56.094522 795 ts_tablet_manager.cc:1939] T d201ebe1653f492e94624c09887335fc P 52f06bd6d6a448aeb07ada8c0d89e2e1: Deleting consensus metadata
I20260806 06:26:56.094812 404 tablet_service.cc:1467] Processing CreateTablet for tablet 62977782ae3b4af1becfad2d4bd056c4 (DEFAULT_TABLE table=test-table [id=a30f3e4e3ed14439af73504568fef3e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 06:26:56.094920 404 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62977782ae3b4af1becfad2d4bd056c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 06:26:56.095293 791 ts_tablet_manager.cc:1918] T d201ebe1653f492e94624c09887335fc P 067b2ec58c34466f99bd66c5780d8fd3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 06:26:56.095341 791 log.cc:1196] T d201ebe1653f492e94624c09887335fc P 067b2ec58c34466f99bd66c5780d8fd3: Deleting WAL directory at /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785997581049705-21554-0/minicluster-data/ts-0/wal/wals/d201ebe1653f492e94624c09887335fc
I20260806 06:26:56.095522 791 ts_tablet_manager.cc:1939] T d201ebe1653f492e94624c09887335fc P 067b2ec58c34466f99bd66c5780d8fd3: Deleting consensus metadata
I20260806 06:26:56.095708 792 ts_tablet_manager.cc:1918] T d201ebe1653f492e94624c09887335fc P 8483d5dffddc4ca080e13001a1707d12: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 06:26:56.095757 792 log.cc:1196] T d201ebe1653f492e94624c09887335fc P 8483d5dffddc4ca080e13001a1707d12: Deleting WAL directory at /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785997581049705-21554-0/minicluster-data/ts-2/wal/wals/d201ebe1653f492e94624c09887335fc
I20260806 06:26:56.095932 792 ts_tablet_manager.cc:1939] T d201ebe1653f492e94624c09887335fc P 8483d5dffddc4ca080e13001a1707d12: Deleting consensus metadata
I20260806 06:26:56.096016 761 tablet_bootstrap.cc:492] T 62977782ae3b4af1becfad2d4bd056c4 P 52f06bd6d6a448aeb07ada8c0d89e2e1: Bootstrap starting.
I20260806 06:26:56.096174 614 tablet_service.cc:1467] Processing CreateTablet for tablet 62977782ae3b4af1becfad2d4bd056c4 (DEFAULT_TABLE table=test-table [id=a30f3e4e3ed14439af73504568fef3e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 06:26:56.096269 614 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62977782ae3b4af1becfad2d4bd056c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 06:26:56.096398 761 tablet_bootstrap.cc:654] T 62977782ae3b4af1becfad2d4bd056c4 P 52f06bd6d6a448aeb07ada8c0d89e2e1: Neither blocks nor log segments found. Creating new log.
I20260806 06:26:56.097327 759 tablet_bootstrap.cc:492] T 62977782ae3b4af1becfad2d4bd056c4 P 8483d5dffddc4ca080e13001a1707d12: Bootstrap starting.
I20260806 06:26:56.097664 759 tablet_bootstrap.cc:654] T 62977782ae3b4af1becfad2d4bd056c4 P 8483d5dffddc4ca080e13001a1707d12: Neither blocks nor log segments found. Creating new log.
I20260806 06:26:56.097966 32407 catalog_manager.cc:4841] TS 067b2ec58c34466f99bd66c5780d8fd3 (127.21.12.129:44983): tablet d201ebe1653f492e94624c09887335fc (table test-table [id=e65089315d3e4c66af237255012c0da6]) successfully deleted
I20260806 06:26:56.098097 32406 catalog_manager.cc:4841] TS 8483d5dffddc4ca080e13001a1707d12 (127.21.12.131:41903): tablet d201ebe1653f492e94624c09887335fc (table test-table [id=e65089315d3e4c66af237255012c0da6]) successfully deleted
I20260806 06:26:56.098162 32406 catalog_manager.cc:4841] TS 52f06bd6d6a448aeb07ada8c0d89e2e1 (127.21.12.130:39591): tablet d201ebe1653f492e94624c09887335fc (table test-table [id=e65089315d3e4c66af237255012c0da6]) successfully deleted
I20260806 06:26:56.098248 759 tablet_bootstrap.cc:492] T 62977782ae3b4af1becfad2d4bd056c4 P 8483d5dffddc4ca080e13001a1707d12: No bootstrap required, opened a new log
I20260806 06:26:56.098309 759 ts_tablet_manager.cc:1397] T 62977782ae3b4af1becfad2d4bd056c4 P 8483d5dffddc4ca080e13001a1707d12: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260806 06:26:56.098423 762 tablet_bootstrap.cc:492] T 62977782ae3b4af1becfad2d4bd056c4 P 067b2ec58c34466f99bd66c5780d8fd3: No bootstrap required, opened a new log
I20260806 06:26:56.098407 759 raft_consensus.cc:348] T 62977782ae3b4af1becfad2d4bd056c4 P 8483d5dffddc4ca080e13001a1707d12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "067b2ec58c34466f99bd66c5780d8fd3" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 44983 } } peers { permanent_uuid: "52f06bd6d6a448aeb07ada8c0d89e2e1" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 39591 } } peers { permanent_uuid: "8483d5dffddc4ca080e13001a1707d12" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 41903 } }
I20260806 06:26:56.098464 759 raft_consensus.cc:374] T 62977782ae3b4af1becfad2d4bd056c4 P 8483d5dffddc4ca080e13001a1707d12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 06:26:56.098464 762 ts_tablet_manager.cc:1397] T 62977782ae3b4af1becfad2d4bd056c4 P 067b2ec58c34466f99bd66c5780d8fd3: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260806 06:26:56.098483 759 raft_consensus.cc:729] T 62977782ae3b4af1becfad2d4bd056c4 P 8483d5dffddc4ca080e13001a1707d12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8483d5dffddc4ca080e13001a1707d12, State: Initialized, Role: FOLLOWER
I20260806 06:26:56.098531 759 consensus_queue.cc:260] T 62977782ae3b4af1becfad2d4bd056c4 P 8483d5dffddc4ca080e13001a1707d12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "067b2ec58c34466f99bd66c5780d8fd3" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 44983 } } peers { permanent_uuid: "52f06bd6d6a448aeb07ada8c0d89e2e1" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 39591 } } peers { permanent_uuid: "8483d5dffddc4ca080e13001a1707d12" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 41903 } }
I20260806 06:26:56.098567 762 raft_consensus.cc:348] T 62977782ae3b4af1becfad2d4bd056c4 P 067b2ec58c34466f99bd66c5780d8fd3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "067b2ec58c34466f99bd66c5780d8fd3" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 44983 } } peers { permanent_uuid: "52f06bd6d6a448aeb07ada8c0d89e2e1" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 39591 } } peers { permanent_uuid: "8483d5dffddc4ca080e13001a1707d12" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 41903 } }
I20260806 06:26:56.098616 759 ts_tablet_manager.cc:1428] T 62977782ae3b4af1becfad2d4bd056c4 P 8483d5dffddc4ca080e13001a1707d12: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260806 06:26:56.098623 762 raft_consensus.cc:374] T 62977782ae3b4af1becfad2d4bd056c4 P 067b2ec58c34466f99bd66c5780d8fd3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 06:26:56.098647 762 raft_consensus.cc:729] T 62977782ae3b4af1becfad2d4bd056c4 P 067b2ec58c34466f99bd66c5780d8fd3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 067b2ec58c34466f99bd66c5780d8fd3, State: Initialized, Role: FOLLOWER
I20260806 06:26:56.098688 762 consensus_queue.cc:260] T 62977782ae3b4af1becfad2d4bd056c4 P 067b2ec58c34466f99bd66c5780d8fd3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "067b2ec58c34466f99bd66c5780d8fd3" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 44983 } } peers { permanent_uuid: "52f06bd6d6a448aeb07ada8c0d89e2e1" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 39591 } } peers { permanent_uuid: "8483d5dffddc4ca080e13001a1707d12" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 41903 } }
I20260806 06:26:56.098773 762 ts_tablet_manager.cc:1428] T 62977782ae3b4af1becfad2d4bd056c4 P 067b2ec58c34466f99bd66c5780d8fd3: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260806 06:26:56.100191 761 tablet_bootstrap.cc:492] T 62977782ae3b4af1becfad2d4bd056c4 P 52f06bd6d6a448aeb07ada8c0d89e2e1: No bootstrap required, opened a new log
I20260806 06:26:56.100250 761 ts_tablet_manager.cc:1397] T 62977782ae3b4af1becfad2d4bd056c4 P 52f06bd6d6a448aeb07ada8c0d89e2e1: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20260806 06:26:56.100410 761 raft_consensus.cc:348] T 62977782ae3b4af1becfad2d4bd056c4 P 52f06bd6d6a448aeb07ada8c0d89e2e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "067b2ec58c34466f99bd66c5780d8fd3" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 44983 } } peers { permanent_uuid: "52f06bd6d6a448aeb07ada8c0d89e2e1" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 39591 } } peers { permanent_uuid: "8483d5dffddc4ca080e13001a1707d12" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 41903 } }
I20260806 06:26:56.100476 761 raft_consensus.cc:374] T 62977782ae3b4af1becfad2d4bd056c4 P 52f06bd6d6a448aeb07ada8c0d89e2e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 06:26:56.100498 761 raft_consensus.cc:729] T 62977782ae3b4af1becfad2d4bd056c4 P 52f06bd6d6a448aeb07ada8c0d89e2e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52f06bd6d6a448aeb07ada8c0d89e2e1, State: Initialized, Role: FOLLOWER
I20260806 06:26:56.100543 761 consensus_queue.cc:260] T 62977782ae3b4af1becfad2d4bd056c4 P 52f06bd6d6a448aeb07ada8c0d89e2e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "067b2ec58c34466f99bd66c5780d8fd3" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 44983 } } peers { permanent_uuid: "52f06bd6d6a448aeb07ada8c0d89e2e1" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 39591 } } peers { permanent_uuid: "8483d5dffddc4ca080e13001a1707d12" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 41903 } }
I20260806 06:26:56.100631 761 ts_tablet_manager.cc:1428] T 62977782ae3b4af1becfad2d4bd056c4 P 52f06bd6d6a448aeb07ada8c0d89e2e1: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260806 06:26:56.110028 768 raft_consensus.cc:482] T 62977782ae3b4af1becfad2d4bd056c4 P 067b2ec58c34466f99bd66c5780d8fd3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 06:26:56.110091 768 raft_consensus.cc:504] T 62977782ae3b4af1becfad2d4bd056c4 P 067b2ec58c34466f99bd66c5780d8fd3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "067b2ec58c34466f99bd66c5780d8fd3" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 44983 } } peers { permanent_uuid: "52f06bd6d6a448aeb07ada8c0d89e2e1" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 39591 } } peers { permanent_uuid: "8483d5dffddc4ca080e13001a1707d12" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 41903 } }
I20260806 06:26:56.110200 768 leader_election.cc:288] T 62977782ae3b4af1becfad2d4bd056c4 P 067b2ec58c34466f99bd66c5780d8fd3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52f06bd6d6a448aeb07ada8c0d89e2e1 (127.21.12.130:39591), 8483d5dffddc4ca080e13001a1707d12 (127.21.12.131:41903)
I20260806 06:26:56.110420 433 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62977782ae3b4af1becfad2d4bd056c4" candidate_uuid: "067b2ec58c34466f99bd66c5780d8fd3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52f06bd6d6a448aeb07ada8c0d89e2e1" is_pre_election: true
I20260806 06:26:56.110436 648 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62977782ae3b4af1becfad2d4bd056c4" candidate_uuid: "067b2ec58c34466f99bd66c5780d8fd3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8483d5dffddc4ca080e13001a1707d12" is_pre_election: true
I20260806 06:26:56.110496 433 raft_consensus.cc:2454] T 62977782ae3b4af1becfad2d4bd056c4 P 52f06bd6d6a448aeb07ada8c0d89e2e1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 067b2ec58c34466f99bd66c5780d8fd3 in term 0.
I20260806 06:26:56.110507 648 raft_consensus.cc:2454] T 62977782ae3b4af1becfad2d4bd056c4 P 8483d5dffddc4ca080e13001a1707d12 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 067b2ec58c34466f99bd66c5780d8fd3 in term 0.
I20260806 06:26:56.110668 32589 leader_election.cc:302] T 62977782ae3b4af1becfad2d4bd056c4 P 067b2ec58c34466f99bd66c5780d8fd3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 067b2ec58c34466f99bd66c5780d8fd3, 8483d5dffddc4ca080e13001a1707d12; no voters:
I20260806 06:26:56.111346 768 raft_consensus.cc:2781] T 62977782ae3b4af1becfad2d4bd056c4 P 067b2ec58c34466f99bd66c5780d8fd3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 06:26:56.111389 768 raft_consensus.cc:482] T 62977782ae3b4af1becfad2d4bd056c4 P 067b2ec58c34466f99bd66c5780d8fd3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 06:26:56.111413 768 raft_consensus.cc:3037] T 62977782ae3b4af1becfad2d4bd056c4 P 067b2ec58c34466f99bd66c5780d8fd3 [term 0 FOLLOWER]: Advancing to term 1
I20260806 06:26:56.112159 768 raft_consensus.cc:504] T 62977782ae3b4af1becfad2d4bd056c4 P 067b2ec58c34466f99bd66c5780d8fd3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "067b2ec58c34466f99bd66c5780d8fd3" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 44983 } } peers { permanent_uuid: "52f06bd6d6a448aeb07ada8c0d89e2e1" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 39591 } } peers { permanent_uuid: "8483d5dffddc4ca080e13001a1707d12" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 41903 } }
I20260806 06:26:56.112275 768 leader_election.cc:288] T 62977782ae3b4af1becfad2d4bd056c4 P 067b2ec58c34466f99bd66c5780d8fd3 [CANDIDATE]: Term 1 election: Requested vote from peers 52f06bd6d6a448aeb07ada8c0d89e2e1 (127.21.12.130:39591), 8483d5dffddc4ca080e13001a1707d12 (127.21.12.131:41903)
I20260806 06:26:56.112452 648 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62977782ae3b4af1becfad2d4bd056c4" candidate_uuid: "067b2ec58c34466f99bd66c5780d8fd3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8483d5dffddc4ca080e13001a1707d12"
I20260806 06:26:56.112454 433 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62977782ae3b4af1becfad2d4bd056c4" candidate_uuid: "067b2ec58c34466f99bd66c5780d8fd3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52f06bd6d6a448aeb07ada8c0d89e2e1"
I20260806 06:26:56.112510 648 raft_consensus.cc:3037] T 62977782ae3b4af1becfad2d4bd056c4 P 8483d5dffddc4ca080e13001a1707d12 [term 0 FOLLOWER]: Advancing to term 1
I20260806 06:26:56.112520 433 raft_consensus.cc:3037] T 62977782ae3b4af1becfad2d4bd056c4 P 52f06bd6d6a448aeb07ada8c0d89e2e1 [term 0 FOLLOWER]: Advancing to term 1
I20260806 06:26:56.113184 648 raft_consensus.cc:2454] T 62977782ae3b4af1becfad2d4bd056c4 P 8483d5dffddc4ca080e13001a1707d12 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 067b2ec58c34466f99bd66c5780d8fd3 in term 1.
I20260806 06:26:56.113291 433 raft_consensus.cc:2454] T 62977782ae3b4af1becfad2d4bd056c4 P 52f06bd6d6a448aeb07ada8c0d89e2e1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 067b2ec58c34466f99bd66c5780d8fd3 in term 1.
I20260806 06:26:56.113371 32589 leader_election.cc:302] T 62977782ae3b4af1becfad2d4bd056c4 P 067b2ec58c34466f99bd66c5780d8fd3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 067b2ec58c34466f99bd66c5780d8fd3, 8483d5dffddc4ca080e13001a1707d12; no voters:
I20260806 06:26:56.113767 768 raft_consensus.cc:2781] T 62977782ae3b4af1becfad2d4bd056c4 P 067b2ec58c34466f99bd66c5780d8fd3 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 06:26:56.113881 768 raft_consensus.cc:686] T 62977782ae3b4af1becfad2d4bd056c4 P 067b2ec58c34466f99bd66c5780d8fd3 [term 1 LEADER]: Becoming Leader. State: Replica: 067b2ec58c34466f99bd66c5780d8fd3, State: Running, Role: LEADER
I20260806 06:26:56.113925 768 consensus_queue.cc:237] T 62977782ae3b4af1becfad2d4bd056c4 P 067b2ec58c34466f99bd66c5780d8fd3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "067b2ec58c34466f99bd66c5780d8fd3" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 44983 } } peers { permanent_uuid: "52f06bd6d6a448aeb07ada8c0d89e2e1" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 39591 } } peers { permanent_uuid: "8483d5dffddc4ca080e13001a1707d12" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 41903 } }
I20260806 06:26:56.114393 32425 catalog_manager.cc:5495] T 62977782ae3b4af1becfad2d4bd056c4 P 067b2ec58c34466f99bd66c5780d8fd3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 067b2ec58c34466f99bd66c5780d8fd3 (127.21.12.129). New cstate: current_term: 1 leader_uuid: "067b2ec58c34466f99bd66c5780d8fd3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "067b2ec58c34466f99bd66c5780d8fd3" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 44983 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52f06bd6d6a448aeb07ada8c0d89e2e1" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 39591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8483d5dffddc4ca080e13001a1707d12" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 41903 } health_report { overall_health: UNKNOWN } } }
I20260806 06:26:56.124396 755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:56.121399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55218 (local address 127.21.12.129:44983)
0806 06:26:56.121437 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0806 06:26:56.121439 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:26:56.121445 (+ 6us) server_negotiation.cc:407] Connection header received
0806 06:26:56.121465 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:26:56.121466 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:26:56.121474 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:26:56.121514 (+ 40us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 06:26:56.121722 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:56.122499 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:56.123569 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:56.123869 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:56.123939 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:56.124078 (+ 139us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 06:26:56.124110 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 06:26:56.124156 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0806 06:26:56.124308 (+ 152us) server_negotiation.cc:299] Negotiation successful
0806 06:26:56.124330 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260806 06:26:56.129680 757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:56.126914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.129:44957 (local address 127.21.12.131:41903)
0806 06:26:56.126951 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0806 06:26:56.126953 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:26:56.126961 (+ 8us) server_negotiation.cc:407] Connection header received
0806 06:26:56.127007 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:26:56.127008 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:26:56.127017 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:26:56.127071 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 06:26:56.127165 (+ 94us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:56.127873 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:56.128941 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:56.129239 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:56.129305 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:56.129461 (+ 156us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 06:26:56.129492 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 06:26:56.129538 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0806 06:26:56.129592 (+ 54us) server_negotiation.cc:299] Negotiation successful
0806 06:26:56.129613 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260806 06:26:56.129710 882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:56.126746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.131:41903 (local address 127.21.12.129:44957)
0806 06:26:56.126889 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0806 06:26:56.126900 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0806 06:26:56.126951 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:26:56.127072 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:26:56.127073 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:26:56.127075 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 06:26:56.127118 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:56.127120 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:56.127943 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:56.127944 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:56.128859 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:56.128862 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:56.129311 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:56.129312 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:56.129369 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:56.129385 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 06:26:56.129545 (+ 160us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 06:26:56.129547 (+ 2us) client_negotiation.cc:769] Sending connection context
0806 06:26:56.129588 (+ 41us) client_negotiation.cc:240] Negotiation successful
0806 06:26:56.129634 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":78,"threads_started":1}
I20260806 06:26:56.132640 648 raft_consensus.cc:1261] T 62977782ae3b4af1becfad2d4bd056c4 P 8483d5dffddc4ca080e13001a1707d12 [term 1 FOLLOWER]: Refusing update from remote peer 067b2ec58c34466f99bd66c5780d8fd3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 06:26:56.132762 433 raft_consensus.cc:1261] T 62977782ae3b4af1becfad2d4bd056c4 P 52f06bd6d6a448aeb07ada8c0d89e2e1 [term 1 FOLLOWER]: Refusing update from remote peer 067b2ec58c34466f99bd66c5780d8fd3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 06:26:56.132954 768 consensus_queue.cc:1035] T 62977782ae3b4af1becfad2d4bd056c4 P 067b2ec58c34466f99bd66c5780d8fd3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52f06bd6d6a448aeb07ada8c0d89e2e1" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 39591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 06:26:56.133086 768 consensus_queue.cc:1035] T 62977782ae3b4af1becfad2d4bd056c4 P 067b2ec58c34466f99bd66c5780d8fd3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8483d5dffddc4ca080e13001a1707d12" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 41903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 06:26:56.142014 755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:56.139194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.131:54021 (local address 127.21.12.129:44983)
0806 06:26:56.139230 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0806 06:26:56.139231 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0806 06:26:56.139239 (+ 8us) server_negotiation.cc:407] Connection header received
0806 06:26:56.139299 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:26:56.139300 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:26:56.139309 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:26:56.139358 (+ 49us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 06:26:56.139484 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:56.140223 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:56.141265 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:56.141558 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:56.141623 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:56.141788 (+ 165us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 06:26:56.141823 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 06:26:56.141873 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0806 06:26:56.141926 (+ 53us) server_negotiation.cc:299] Negotiation successful
0806 06:26:56.141949 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260806 06:26:56.142055 832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:56.139101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.129:44983 (local address 127.21.12.131:54021)
0806 06:26:56.139198 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0806 06:26:56.139208 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0806 06:26:56.139246 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:26:56.139371 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:26:56.139372 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:26:56.139373 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 06:26:56.139412 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:56.139414 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:56.140299 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:56.140300 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:56.141188 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:56.141191 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:56.141659 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:56.141660 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:56.141720 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:56.141727 (+ 7us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 06:26:56.141882 (+ 155us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 06:26:56.141884 (+ 2us) client_negotiation.cc:769] Sending connection context
0806 06:26:56.141926 (+ 42us) client_negotiation.cc:240] Negotiation successful
0806 06:26:56.141971 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":77}
I20260806 06:26:56.152139 32425 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33178:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 06:26:56.152244 32425 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33178:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 06:26:56.152935 32425 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 987ca9a955c54e639f87808644890c4c: Sending DeleteTablet for 3 replicas of tablet 62977782ae3b4af1becfad2d4bd056c4
I20260806 06:26:56.153218 404 tablet_service.cc:1514] Processing DeleteTablet for tablet 62977782ae3b4af1becfad2d4bd056c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 06:26:56 UTC) from {username='oryx', principal='oryx/127.21.12.190@KRBTEST.COM'} at 127.0.0.1:39336
I20260806 06:26:56.153313 795 tablet_replica.cc:330] T 62977782ae3b4af1becfad2d4bd056c4 P 52f06bd6d6a448aeb07ada8c0d89e2e1: stopping tablet replica
I20260806 06:26:56.153355 795 raft_consensus.cc:2229] T 62977782ae3b4af1becfad2d4bd056c4 P 52f06bd6d6a448aeb07ada8c0d89e2e1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 06:26:56.153388 795 raft_consensus.cc:2258] T 62977782ae3b4af1becfad2d4bd056c4 P 52f06bd6d6a448aeb07ada8c0d89e2e1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 06:26:56.153448 614 tablet_service.cc:1514] Processing DeleteTablet for tablet 62977782ae3b4af1becfad2d4bd056c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 06:26:56 UTC) from {username='oryx', principal='oryx/127.21.12.190@KRBTEST.COM'} at 127.0.0.1:44776
I20260806 06:26:56.153522 792 tablet_replica.cc:330] T 62977782ae3b4af1becfad2d4bd056c4 P 8483d5dffddc4ca080e13001a1707d12: stopping tablet replica
I20260806 06:26:56.153571 792 raft_consensus.cc:2229] T 62977782ae3b4af1becfad2d4bd056c4 P 8483d5dffddc4ca080e13001a1707d12 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 06:26:56.153600 792 raft_consensus.cc:2258] T 62977782ae3b4af1becfad2d4bd056c4 P 8483d5dffddc4ca080e13001a1707d12 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 06:26:56.153805 32651 tablet_service.cc:1514] Processing DeleteTablet for tablet 62977782ae3b4af1becfad2d4bd056c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 06:26:56 UTC) from {username='oryx', principal='oryx/127.21.12.190@KRBTEST.COM'} at 127.0.0.1:55198
I20260806 06:26:56.153910 791 tablet_replica.cc:330] T 62977782ae3b4af1becfad2d4bd056c4 P 067b2ec58c34466f99bd66c5780d8fd3: stopping tablet replica
I20260806 06:26:56.153990 791 raft_consensus.cc:2229] T 62977782ae3b4af1becfad2d4bd056c4 P 067b2ec58c34466f99bd66c5780d8fd3 [term 1 LEADER]: Raft consensus shutting down.
I20260806 06:26:56.154060 791 raft_consensus.cc:2258] T 62977782ae3b4af1becfad2d4bd056c4 P 067b2ec58c34466f99bd66c5780d8fd3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 06:26:56.154420 791 ts_tablet_manager.cc:1905] T 62977782ae3b4af1becfad2d4bd056c4 P 067b2ec58c34466f99bd66c5780d8fd3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 06:26:56.154500 21554 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHhLhT/build/release/bin/kudu with pid 32555
I20260806 06:26:56.162740 792 ts_tablet_manager.cc:1905] T 62977782ae3b4af1becfad2d4bd056c4 P 8483d5dffddc4ca080e13001a1707d12: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 06:26:56.164362 792 ts_tablet_manager.cc:1918] T 62977782ae3b4af1becfad2d4bd056c4 P 8483d5dffddc4ca080e13001a1707d12: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 06:26:56.164422 792 log.cc:1196] T 62977782ae3b4af1becfad2d4bd056c4 P 8483d5dffddc4ca080e13001a1707d12: Deleting WAL directory at /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785997581049705-21554-0/minicluster-data/ts-2/wal/wals/62977782ae3b4af1becfad2d4bd056c4
I20260806 06:26:56.164441 795 ts_tablet_manager.cc:1905] T 62977782ae3b4af1becfad2d4bd056c4 P 52f06bd6d6a448aeb07ada8c0d89e2e1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 06:26:56.164633 792 ts_tablet_manager.cc:1939] T 62977782ae3b4af1becfad2d4bd056c4 P 8483d5dffddc4ca080e13001a1707d12: Deleting consensus metadata
I20260806 06:26:56.165809 795 ts_tablet_manager.cc:1918] T 62977782ae3b4af1becfad2d4bd056c4 P 52f06bd6d6a448aeb07ada8c0d89e2e1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 06:26:56.165868 795 log.cc:1196] T 62977782ae3b4af1becfad2d4bd056c4 P 52f06bd6d6a448aeb07ada8c0d89e2e1: Deleting WAL directory at /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785997581049705-21554-0/minicluster-data/ts-1/wal/wals/62977782ae3b4af1becfad2d4bd056c4
I20260806 06:26:56.166067 795 ts_tablet_manager.cc:1939] T 62977782ae3b4af1becfad2d4bd056c4 P 52f06bd6d6a448aeb07ada8c0d89e2e1: Deleting consensus metadata
I20260806 06:26:56.166447 32406 catalog_manager.cc:4841] TS 52f06bd6d6a448aeb07ada8c0d89e2e1 (127.21.12.130:39591): tablet 62977782ae3b4af1becfad2d4bd056c4 (table test-table [id=a30f3e4e3ed14439af73504568fef3e4]) successfully deleted
I20260806 06:26:56.166657 32406 catalog_manager.cc:4841] TS 8483d5dffddc4ca080e13001a1707d12 (127.21.12.131:41903): tablet 62977782ae3b4af1becfad2d4bd056c4 (table test-table [id=a30f3e4e3ed14439af73504568fef3e4]) successfully deleted
W20260806 06:26:56.168182 32407 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.12.129:44983 (error 108)
W20260806 06:26:56.168887 889 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 06:26:56.168595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.129:44983 (local address 127.0.0.1:55222)
0806 06:26:56.168698 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0806 06:26:56.168711 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 06:26:56.168770 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:26:56.168839 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.129:44983: BlockingWrite error: write error: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":83,"thread_start_us":49,"threads_started":1}
W20260806 06:26:56.169101 32407 catalog_manager.cc:4584] TS 067b2ec58c34466f99bd66c5780d8fd3 (127.21.12.129:44983): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 62977782ae3b4af1becfad2d4bd056c4: Network error: Client connection negotiation failed: client connection to 127.21.12.129:44983: BlockingWrite error: write error: Connection reset by peer (error 104)
I20260806 06:26:56.169471 21554 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHhLhT/build/release/bin/kudu with pid 304
I20260806 06:26:56.180951 21554 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHhLhT/build/release/bin/kudu with pid 518
I20260806 06:26:56.191653 21554 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHhLhT/build/release/bin/kudu with pid 32378
2026-08-06T06:26:56Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (2639 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785997581049705-21554-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 06:26:56 dist-test-slave-4tdj krb5kdc[894](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 06:26:56 dist-test-slave-4tdj krb5kdc[894](info): set up 2 sockets
Aug 06 06:26:56 dist-test-slave-4tdj krb5kdc[894](info): commencing operation
krb5kdc: starting...
W20260806 06:26:57.305457 21554 mini_kdc.cc:121] Time spent starting KDC: real 1.088s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785997581049705-21554-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785997581049705-21554-0/krb5kdc/test-user.keytab.
Aug 06 06:26:57 dist-test-slave-4tdj krb5kdc[894](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997617, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T06:26:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T06:26:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.12.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.12.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.12.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.190.keytab.
Entry for principal kudu/127.21.12.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.190.keytab.
I20260806 06:26:57.417248 21554 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHhLhT/build/release/bin/kudu
/tmp/dist-test-taskFHhLhT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785997581049705-21554-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785997581049705-21554-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785997581049705-21554-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785997581049705-21554-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.12.190:46391
--webserver_interface=127.21.12.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.190.keytab
--principal=kudu/127.21.12.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.148:45125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.12.190:46391
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785997581049705-21554-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785997581049705-21554-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785997581049705-21554-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 06:26:57.497574 917 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 06:26:57.497763 917 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 06:26:57.497783 917 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 06:26:57.498883 917 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 06:26:57.498916 917 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 06:26:57.498929 917 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 06:26:57.498939 917 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 06:26:57.498950 917 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 06:26:57.500352 917 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.148:45125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785997581049705-21554-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785997581049705-21554-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.12.190:46391
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.190.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.12.190:46391
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785997581049705-21554-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.12.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.917
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785997581049705-21554-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 05:11:02 UTC on ccycloud.quasar-hvzbbz.root.comops.site
build id 4569617
I20260806 06:26:57.500540 917 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 06:26:57.500699 917 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 06:26:57.505239 926 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:26:57.505273 923 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:26:57.505266 924 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:26:57.505460 917 server_base.cc:1034] running on GCE node
I20260806 06:26:57.505683 917 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 06:26:57.505935 917 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 06:26:57.507066 917 hybrid_clock.cc:648] HybridClock initialized: now 1785997617507052 us; error 27 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-08-06T06:26:57Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1316 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785997581049705-21554-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 06:26:57 dist-test-slave-4tdj krb5kdc[931](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 06:26:57 dist-test-slave-4tdj krb5kdc[931](info): set up 2 sockets
Aug 06 06:26:57 dist-test-slave-4tdj krb5kdc[931](info): commencing operation
krb5kdc: starting...
W20260806 06:26:58.784417 21554 mini_kdc.cc:121] Time spent starting KDC: real 1.250s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785997581049705-21554-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785997581049705-21554-0/krb5kdc/test-user.keytab.
Aug 06 06:26:58 dist-test-slave-4tdj krb5kdc[931](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997618, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T06:26:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T06:26:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.12.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.12.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.12.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.190.keytab.
Entry for principal kudu/127.21.12.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.190.keytab.
I20260806 06:26:58.955755 21554 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHhLhT/build/release/bin/kudu
/tmp/dist-test-taskFHhLhT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785997581049705-21554-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785997581049705-21554-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785997581049705-21554-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785997581049705-21554-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.12.190:46519
--webserver_interface=127.21.12.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.190.keytab
--principal=kudu/127.21.12.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.148:37101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.12.190:46519
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785997581049705-21554-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785997581049705-21554-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785997581049705-21554-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 06:26:59.104739 955 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 06:26:59.105101 955 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 06:26:59.105161 955 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 06:26:59.107096 955 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 06:26:59.107158 955 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 06:26:59.107209 955 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 06:26:59.107255 955 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 06:26:59.107290 955 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 06:26:59.109807 955 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.148:37101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785997581049705-21554-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785997581049705-21554-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.12.190:46519
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.190.keytab
--principal=kudu/127.21.12.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.12.190:46519
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785997581049705-21554-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.12.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.955
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785997581049705-21554-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 05:11:02 UTC on ccycloud.quasar-hvzbbz.root.comops.site
build id 4569617
I20260806 06:26:59.110224 955 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 06:26:59.110707 955 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 06:26:59.118494 962 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:26:59.118539 967 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:26:59.118508 961 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:26:59.118939 955 server_base.cc:1034] running on GCE node
I20260806 06:26:59.119300 955 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 06:26:59.119714 955 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 06:26:59.120924 955 hybrid_clock.cc:648] HybridClock initialized: now 1785997619120891 us; error 40 us; skew 500 ppm
Aug 06 06:26:59 dist-test-slave-4tdj krb5kdc[931](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997619, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 06:26:59.125240 955 init.cc:377] Logged in from keytab as kudu/127.21.12.190@KRBTEST.COM (short username kudu)
I20260806 06:26:59.126032 955 webserver.cc:460] Webserver started at http://127.21.12.190:35413/ using document root <none> and password file <none>
I20260806 06:26:59.126461 955 fs_manager.cc:359] Metadata directory not provided
I20260806 06:26:59.126590 955 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 06:26:59.126840 955 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 06:26:59.128630 955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785997581049705-21554-0/minicluster-data/master-0/data/instance:
uuid: "980b938cd96e4402bf5882896a7bc3eb"
format_stamp: "Formatted at 2026-08-06 06:26:59 on dist-test-slave-4tdj"
server_key: "e1d4da835657ccf9d8d7cf0130b5564e"
server_key_iv: "d32f2e2ca24679ea7013801fc6e1be40"
server_key_version: "encryptionkey@0"
I20260806 06:26:59.129208 955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785997581049705-21554-0/minicluster-data/master-0/wal/instance:
uuid: "980b938cd96e4402bf5882896a7bc3eb"
format_stamp: "Formatted at 2026-08-06 06:26:59 on dist-test-slave-4tdj"
server_key: "e1d4da835657ccf9d8d7cf0130b5564e"
server_key_iv: "d32f2e2ca24679ea7013801fc6e1be40"
server_key_version: "encryptionkey@0"
I20260806 06:26:59.131624 955 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.004s
I20260806 06:26:59.132854 980 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 06:26:59.133116 955 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 06:26:59.133275 955 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785997581049705-21554-0/minicluster-data/master-0/data,/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785997581049705-21554-0/minicluster-data/master-0/wal
uuid: "980b938cd96e4402bf5882896a7bc3eb"
format_stamp: "Formatted at 2026-08-06 06:26:59 on dist-test-slave-4tdj"
server_key: "e1d4da835657ccf9d8d7cf0130b5564e"
server_key_iv: "d32f2e2ca24679ea7013801fc6e1be40"
server_key_version: "encryptionkey@0"
I20260806 06:26:59.133365 955 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785997581049705-21554-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785997581049705-21554-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785997581049705-21554-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 06:26:59.158679 955 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 06:26:59.162173 955 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 06:26:59.162454 955 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 06:26:59.169445 955 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.190:46519
I20260806 06:26:59.169500 1061 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.190:46519 every 8 connection(s)
I20260806 06:26:59.170066 955 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785997581049705-21554-0/minicluster-data/master-0/data/info.pb
I20260806 06:26:59.171663 1062 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 06:26:59.173301 21554 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHhLhT/build/release/bin/kudu as pid 955
I20260806 06:26:59.173501 21554 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785997581049705-21554-0/minicluster-data/master-0/wal/instance
I20260806 06:26:59.173875 21554 external_mini_cluster.cc:1428] Setting key cbfef0a97c7de6d3f2fde52b1a9f7c64
I20260806 06:26:59.176200 1062 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 980b938cd96e4402bf5882896a7bc3eb: Bootstrap starting.
I20260806 06:26:59.177227 1062 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 980b938cd96e4402bf5882896a7bc3eb: Neither blocks nor log segments found. Creating new log.
I20260806 06:26:59.177727 1062 log.cc:824] T 00000000000000000000000000000000 P 980b938cd96e4402bf5882896a7bc3eb: Log is configured to *not* fsync() on all Append() calls
I20260806 06:26:59.179267 1062 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 980b938cd96e4402bf5882896a7bc3eb: No bootstrap required, opened a new log
I20260806 06:26:59.182286 1062 raft_consensus.cc:348] T 00000000000000000000000000000000 P 980b938cd96e4402bf5882896a7bc3eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "980b938cd96e4402bf5882896a7bc3eb" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 46519 } }
I20260806 06:26:59.182605 1062 raft_consensus.cc:374] T 00000000000000000000000000000000 P 980b938cd96e4402bf5882896a7bc3eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 06:26:59.182694 1062 raft_consensus.cc:729] T 00000000000000000000000000000000 P 980b938cd96e4402bf5882896a7bc3eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 980b938cd96e4402bf5882896a7bc3eb, State: Initialized, Role: FOLLOWER
I20260806 06:26:59.182924 1062 consensus_queue.cc:260] T 00000000000000000000000000000000 P 980b938cd96e4402bf5882896a7bc3eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "980b938cd96e4402bf5882896a7bc3eb" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 46519 } }
I20260806 06:26:59.183044 1062 raft_consensus.cc:388] T 00000000000000000000000000000000 P 980b938cd96e4402bf5882896a7bc3eb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 06:26:59.183128 1062 raft_consensus.cc:482] T 00000000000000000000000000000000 P 980b938cd96e4402bf5882896a7bc3eb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 06:26:59.183225 1062 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 980b938cd96e4402bf5882896a7bc3eb [term 0 FOLLOWER]: Advancing to term 1
I20260806 06:26:59.184206 1062 raft_consensus.cc:504] T 00000000000000000000000000000000 P 980b938cd96e4402bf5882896a7bc3eb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "980b938cd96e4402bf5882896a7bc3eb" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 46519 } }
I20260806 06:26:59.184386 1062 leader_election.cc:302] T 00000000000000000000000000000000 P 980b938cd96e4402bf5882896a7bc3eb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 980b938cd96e4402bf5882896a7bc3eb; no voters:
I20260806 06:26:59.184713 1062 leader_election.cc:288] T 00000000000000000000000000000000 P 980b938cd96e4402bf5882896a7bc3eb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 06:26:59.184809 1072 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 980b938cd96e4402bf5882896a7bc3eb [term 1 FOLLOWER]: Leader election won for term 1
Aug 06 06:26:59 dist-test-slave-4tdj krb5kdc[931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997618, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.12.190@KRBTEST.COM
I20260806 06:26:59.185086 1072 raft_consensus.cc:686] T 00000000000000000000000000000000 P 980b938cd96e4402bf5882896a7bc3eb [term 1 LEADER]: Becoming Leader. State: Replica: 980b938cd96e4402bf5882896a7bc3eb, State: Running, Role: LEADER
I20260806 06:26:59.185240 1062 sys_catalog.cc:564] T 00000000000000000000000000000000 P 980b938cd96e4402bf5882896a7bc3eb [sys.catalog]: configured and running, proceeding with master startup.
I20260806 06:26:59.185285 1072 consensus_queue.cc:237] T 00000000000000000000000000000000 P 980b938cd96e4402bf5882896a7bc3eb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "980b938cd96e4402bf5882896a7bc3eb" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 46519 } }
I20260806 06:26:59.185948 1073 sys_catalog.cc:455] T 00000000000000000000000000000000 P 980b938cd96e4402bf5882896a7bc3eb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "980b938cd96e4402bf5882896a7bc3eb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "980b938cd96e4402bf5882896a7bc3eb" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 46519 } } }
I20260806 06:26:59.185966 1074 sys_catalog.cc:455] T 00000000000000000000000000000000 P 980b938cd96e4402bf5882896a7bc3eb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 980b938cd96e4402bf5882896a7bc3eb. Latest consensus state: current_term: 1 leader_uuid: "980b938cd96e4402bf5882896a7bc3eb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "980b938cd96e4402bf5882896a7bc3eb" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 46519 } } }
I20260806 06:26:59.186151 1073 sys_catalog.cc:458] T 00000000000000000000000000000000 P 980b938cd96e4402bf5882896a7bc3eb [sys.catalog]: This master's current role is: LEADER
I20260806 06:26:59.186175 1074 sys_catalog.cc:458] T 00000000000000000000000000000000 P 980b938cd96e4402bf5882896a7bc3eb [sys.catalog]: This master's current role is: LEADER
I20260806 06:26:59.187613 1079 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 06:26:59.188557 1079 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 06:26:59.191480 1079 catalog_manager.cc:1347] Generated new cluster ID: a0474fe8137641c28a8fbe8035e05786
I20260806 06:26:59.191566 1079 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 06:26:59.191901 1066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:59.174675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38442 (local address 127.21.12.190:46519)
0806 06:26:59.174990 (+ 315us) server_negotiation.cc:206] Beginning negotiation
0806 06:26:59.175028 (+ 38us) server_negotiation.cc:399] Waiting for connection header
0806 06:26:59.175074 (+ 46us) server_negotiation.cc:407] Connection header received
0806 06:26:59.175903 (+ 829us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:26:59.175915 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:26:59.176000 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:26:59.176220 (+ 220us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:26:59.177330 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:59.179351 (+ 2021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:59.180672 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:59.181377 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:59.181568 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:59.186126 (+ 4558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:26:59.186142 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:26:59.186142 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 06:26:59.186164 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:26:59.189351 (+ 3187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:26:59.189891 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:26:59.189894 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:26:59.189894 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:26:59.190024 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:26:59.190413 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:26:59.190414 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:26:59.190414 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:26:59.190951 (+ 537us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:26:59.191189 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0806 06:26:59.191388 (+ 199us) server_negotiation.cc:299] Negotiation successful
0806 06:26:59.191651 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":231,"thread_start_us":143,"threads_started":1}
I20260806 06:26:59.227206 1079 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 06:26:59.228600 1079 catalog_manager.cc:1504] Loading token signing keys...
I20260806 06:26:59.240568 1079 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 980b938cd96e4402bf5882896a7bc3eb: Generated new TSK 0
I20260806 06:26:59.240942 1079 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.12.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.12.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.12.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.129.keytab.
Entry for principal kudu/127.21.12.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.129.keytab.
I20260806 06:26:59.294782 21554 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHhLhT/build/release/bin/kudu
/tmp/dist-test-taskFHhLhT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785997581049705-21554-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785997581049705-21554-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785997581049705-21554-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785997581049705-21554-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.12.129:0
--local_ip_for_outbound_sockets=127.21.12.129
--webserver_interface=127.21.12.129
--webserver_port=0
--tserver_master_addrs=127.21.12.190:46519
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.129.keytab
--principal=kudu/127.21.12.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.148:37101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785997581049705-21554-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785997581049705-21554-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785997581049705-21554-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 06:26:59.453330 1104 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 06:26:59.453743 1104 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 06:26:59.453825 1104 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 06:26:59.455744 1104 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 06:26:59.455816 1104 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 06:26:59.455891 1104 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.12.129
I20260806 06:26:59.458236 1104 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.148:37101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785997581049705-21554-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785997581049705-21554-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.129.keytab
--principal=kudu/127.21.12.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.12.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785997581049705-21554-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.12.129
--webserver_port=0
--tserver_master_addrs=127.21.12.190:46519
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1104
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.12.129
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785997581049705-21554-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 05:11:02 UTC on ccycloud.quasar-hvzbbz.root.comops.site
build id 4569617
I20260806 06:26:59.458693 1104 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 06:26:59.459115 1104 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 06:26:59.468772 1115 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:26:59.468779 1116 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:26:59.469065 1118 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:26:59.469453 1104 server_base.cc:1034] running on GCE node
I20260806 06:26:59.469832 1104 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 06:26:59.470204 1104 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 06:26:59.471392 1104 hybrid_clock.cc:648] HybridClock initialized: now 1785997619471378 us; error 74 us; skew 500 ppm
Aug 06 06:26:59 dist-test-slave-4tdj krb5kdc[931](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997619, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 06:26:59.475695 1104 init.cc:377] Logged in from keytab as kudu/127.21.12.129@KRBTEST.COM (short username kudu)
I20260806 06:26:59.476500 1104 webserver.cc:460] Webserver started at http://127.21.12.129:33681/ using document root <none> and password file <none>
I20260806 06:26:59.477092 1104 fs_manager.cc:359] Metadata directory not provided
I20260806 06:26:59.477219 1104 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 06:26:59.477525 1104 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 06:26:59.479748 1104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785997581049705-21554-0/minicluster-data/ts-0/data/instance:
uuid: "8d4e9220c4c544d9b3804aa46a8da5e2"
format_stamp: "Formatted at 2026-08-06 06:26:59 on dist-test-slave-4tdj"
server_key: "a62215c8febf74b08f8705ed1186a69d"
server_key_iv: "92202797ec66aa228bafee9615ac93af"
server_key_version: "encryptionkey@0"
I20260806 06:26:59.480463 1104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785997581049705-21554-0/minicluster-data/ts-0/wal/instance:
uuid: "8d4e9220c4c544d9b3804aa46a8da5e2"
format_stamp: "Formatted at 2026-08-06 06:26:59 on dist-test-slave-4tdj"
server_key: "a62215c8febf74b08f8705ed1186a69d"
server_key_iv: "92202797ec66aa228bafee9615ac93af"
server_key_version: "encryptionkey@0"
I20260806 06:26:59.483098 1104 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.004s
I20260806 06:26:59.484265 1130 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 06:26:59.484488 1104 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260806 06:26:59.484649 1104 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785997581049705-21554-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785997581049705-21554-0/minicluster-data/ts-0/wal
uuid: "8d4e9220c4c544d9b3804aa46a8da5e2"
format_stamp: "Formatted at 2026-08-06 06:26:59 on dist-test-slave-4tdj"
server_key: "a62215c8febf74b08f8705ed1186a69d"
server_key_iv: "92202797ec66aa228bafee9615ac93af"
server_key_version: "encryptionkey@0"
I20260806 06:26:59.484746 1104 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785997581049705-21554-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785997581049705-21554-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785997581049705-21554-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 06:26:59.499106 1104 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 06:26:59.502173 1104 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 06:26:59.502395 1104 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 06:26:59.502810 1104 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 06:26:59.503285 1104 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 06:26:59.503340 1104 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 06:26:59.503382 1104 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 06:26:59.503424 1104 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 06:26:59.513535 1104 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.129:44089
I20260806 06:26:59.513628 1297 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.129:44089 every 8 connection(s)
I20260806 06:26:59.514112 1104 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785997581049705-21554-0/minicluster-data/ts-0/data/info.pb
I20260806 06:26:59.516150 21554 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHhLhT/build/release/bin/kudu as pid 1104
I20260806 06:26:59.516300 21554 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785997581049705-21554-0/minicluster-data/ts-0/wal/instance
I20260806 06:26:59.516618 21554 external_mini_cluster.cc:1428] Setting key 8c083fe2d4955e9aa5ad2fc73bac8cb7
Aug 06 06:26:59 dist-test-slave-4tdj krb5kdc[931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997619, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.129@KRBTEST.COM for kudu/127.21.12.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.21.12.130@KRBTEST.COM; defaulting to no policy
I20260806 06:26:59.531041 1066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:59.515301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.129:47159 (local address 127.21.12.190:46519)
0806 06:26:59.515405 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0806 06:26:59.515409 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 06:26:59.515710 (+ 301us) server_negotiation.cc:407] Connection header received
0806 06:26:59.516589 (+ 879us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:26:59.516590 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:26:59.516614 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:26:59.516707 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:26:59.518916 (+ 2209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:59.520158 (+ 1242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:59.521792 (+ 1634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:59.522149 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:59.522263 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:59.526393 (+ 4130us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:26:59.526395 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:26:59.526395 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 06:26:59.526411 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:26:59.528528 (+ 2117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:26:59.529290 (+ 762us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:26:59.529291 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:26:59.529292 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:26:59.529363 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:26:59.529849 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:26:59.529850 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:26:59.529850 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:26:59.530023 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:26:59.530102 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0806 06:26:59.530813 (+ 711us) server_negotiation.cc:299] Negotiation successful
0806 06:26:59.530936 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260806 06:26:59.531522 1301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:59.515270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:46519 (local address 127.21.12.129:47159)
0806 06:26:59.515524 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0806 06:26:59.515592 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0806 06:26:59.516418 (+ 826us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:26:59.516834 (+ 416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:26:59.516850 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:26:59.518092 (+ 1242us) client_negotiation.cc:189] Negotiated authn=SASL
0806 06:26:59.518733 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:59.518753 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:59.520294 (+ 1541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:59.520295 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:59.521677 (+ 1382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:59.521679 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:59.522276 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:59.522277 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:59.522564 (+ 287us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:59.523337 (+ 773us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 06:26:59.523354 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 06:26:59.526168 (+ 2814us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 06:26:59.528654 (+ 2486us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:26:59.528672 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:26:59.528675 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:26:59.529190 (+ 515us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:26:59.529452 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:26:59.529453 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:26:59.529453 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:26:59.529723 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:26:59.530115 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 06:26:59.530123 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 06:26:59.530549 (+ 426us) client_negotiation.cc:769] Sending connection context
0806 06:26:59.530766 (+ 217us) client_negotiation.cc:240] Negotiation successful
0806 06:26:59.531073 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":120,"threads_started":1}
I20260806 06:26:59.532308 1298 heartbeater.cc:347] Connected to a master server at 127.21.12.190:46519
I20260806 06:26:59.532459 1298 heartbeater.cc:464] Registering TS with master...
I20260806 06:26:59.532765 1298 heartbeater.cc:511] Master 127.21.12.190:46519 requested a full tablet report, sending...
I20260806 06:26:59.533557 1004 ts_manager.cc:194] Registered new tserver with Master: 8d4e9220c4c544d9b3804aa46a8da5e2 (127.21.12.129:44089)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.12.130@KRBTEST.COM" created.
I20260806 06:26:59.534835 1004 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.12.129@KRBTEST.COM'} at 127.21.12.129:47159
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.12.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.130.keytab.
Entry for principal kudu/127.21.12.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.130.keytab.
I20260806 06:26:59.554121 21554 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHhLhT/build/release/bin/kudu
/tmp/dist-test-taskFHhLhT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785997581049705-21554-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785997581049705-21554-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785997581049705-21554-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785997581049705-21554-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.12.130:0
--local_ip_for_outbound_sockets=127.21.12.130
--webserver_interface=127.21.12.130
--webserver_port=0
--tserver_master_addrs=127.21.12.190:46519
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.130.keytab
--principal=kudu/127.21.12.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.148:37101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785997581049705-21554-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785997581049705-21554-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785997581049705-21554-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 06:26:59.695214 1308 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 06:26:59.695592 1308 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 06:26:59.695686 1308 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 06:26:59.697553 1308 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 06:26:59.697631 1308 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 06:26:59.697707 1308 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.12.130
I20260806 06:26:59.699950 1308 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.148:37101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785997581049705-21554-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785997581049705-21554-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.130.keytab
--principal=kudu/127.21.12.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.12.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785997581049705-21554-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.12.130
--webserver_port=0
--tserver_master_addrs=127.21.12.190:46519
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1308
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.12.130
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785997581049705-21554-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 05:11:02 UTC on ccycloud.quasar-hvzbbz.root.comops.site
build id 4569617
I20260806 06:26:59.700433 1308 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 06:26:59.700860 1308 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 06:26:59.708699 1321 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:26:59.708719 1319 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:26:59.708742 1324 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:26:59.709280 1308 server_base.cc:1034] running on GCE node
I20260806 06:26:59.709591 1308 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 06:26:59.710000 1308 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 06:26:59.711189 1308 hybrid_clock.cc:648] HybridClock initialized: now 1785997619711172 us; error 40 us; skew 500 ppm
Aug 06 06:26:59 dist-test-slave-4tdj krb5kdc[931](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997619, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 06:26:59.715226 1308 init.cc:377] Logged in from keytab as kudu/127.21.12.130@KRBTEST.COM (short username kudu)
I20260806 06:26:59.715919 1308 webserver.cc:460] Webserver started at http://127.21.12.130:40207/ using document root <none> and password file <none>
I20260806 06:26:59.716238 1308 fs_manager.cc:359] Metadata directory not provided
I20260806 06:26:59.716320 1308 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 06:26:59.716490 1308 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 06:26:59.718256 1308 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785997581049705-21554-0/minicluster-data/ts-1/data/instance:
uuid: "55552f6bc0d64fceb9de0589f1379083"
format_stamp: "Formatted at 2026-08-06 06:26:59 on dist-test-slave-4tdj"
server_key: "bf2d09065f906035b9b65fd2da5a2c25"
server_key_iv: "05bc261fec486115c38e31e54fa460d6"
server_key_version: "encryptionkey@0"
I20260806 06:26:59.718853 1308 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785997581049705-21554-0/minicluster-data/ts-1/wal/instance:
uuid: "55552f6bc0d64fceb9de0589f1379083"
format_stamp: "Formatted at 2026-08-06 06:26:59 on dist-test-slave-4tdj"
server_key: "bf2d09065f906035b9b65fd2da5a2c25"
server_key_iv: "05bc261fec486115c38e31e54fa460d6"
server_key_version: "encryptionkey@0"
I20260806 06:26:59.720870 1308 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260806 06:26:59.722084 1336 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 06:26:59.722364 1308 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 06:26:59.722496 1308 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785997581049705-21554-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785997581049705-21554-0/minicluster-data/ts-1/wal
uuid: "55552f6bc0d64fceb9de0589f1379083"
format_stamp: "Formatted at 2026-08-06 06:26:59 on dist-test-slave-4tdj"
server_key: "bf2d09065f906035b9b65fd2da5a2c25"
server_key_iv: "05bc261fec486115c38e31e54fa460d6"
server_key_version: "encryptionkey@0"
I20260806 06:26:59.722590 1308 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785997581049705-21554-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785997581049705-21554-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785997581049705-21554-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 06:26:59.755070 1308 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 06:26:59.758435 1308 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 06:26:59.758620 1308 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 06:26:59.758940 1308 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 06:26:59.759450 1308 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 06:26:59.759536 1308 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 06:26:59.759599 1308 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 06:26:59.759639 1308 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 06:26:59.769815 1308 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.130:39719
I20260806 06:26:59.769941 1512 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.130:39719 every 8 connection(s)
I20260806 06:26:59.770452 1308 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785997581049705-21554-0/minicluster-data/ts-1/data/info.pb
I20260806 06:26:59.772166 21554 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHhLhT/build/release/bin/kudu as pid 1308
I20260806 06:26:59.772312 21554 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785997581049705-21554-0/minicluster-data/ts-1/wal/instance
I20260806 06:26:59.772519 21554 external_mini_cluster.cc:1428] Setting key 9507232c75ba4a1f939c75f8f070060f
Aug 06 06:26:59 dist-test-slave-4tdj krb5kdc[931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997619, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.130@KRBTEST.COM for kudu/127.21.12.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.21.12.131@KRBTEST.COM; defaulting to no policy
I20260806 06:26:59.785733 1066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:59.771652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.130:57663 (local address 127.21.12.190:46519)
0806 06:26:59.771733 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0806 06:26:59.771744 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 06:26:59.772209 (+ 465us) server_negotiation.cc:407] Connection header received
0806 06:26:59.772977 (+ 768us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:26:59.772979 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:26:59.772994 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:26:59.773094 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:26:59.775441 (+ 2347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:59.776153 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:59.777611 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:26:59.777907 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:26:59.777995 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:59.781375 (+ 3380us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:26:59.781386 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:26:59.781386 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 06:26:59.781394 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:26:59.783641 (+ 2247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:26:59.784228 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:26:59.784229 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:26:59.784229 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:26:59.784335 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:26:59.784687 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:26:59.784688 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:26:59.784689 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:26:59.784850 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:26:59.784943 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0806 06:26:59.785359 (+ 416us) server_negotiation.cc:299] Negotiation successful
0806 06:26:59.785518 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260806 06:26:59.786018 1517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:26:59.771668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:46519 (local address 127.21.12.130:57663)
0806 06:26:59.771972 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0806 06:26:59.772035 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0806 06:26:59.772844 (+ 809us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:26:59.773200 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:26:59.773215 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:26:59.774714 (+ 1499us) client_negotiation.cc:189] Negotiated authn=SASL
0806 06:26:59.775299 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:59.775308 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:59.776308 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:59.776308 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:59.777498 (+ 1190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:26:59.777501 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:26:59.778010 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:26:59.778019 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:26:59.778254 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:26:59.779100 (+ 846us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 06:26:59.779110 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0806 06:26:59.781237 (+ 2127us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 06:26:59.783741 (+ 2504us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:26:59.783749 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:26:59.783760 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:26:59.784119 (+ 359us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:26:59.784410 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:26:59.784420 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:26:59.784420 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:26:59.784584 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:26:59.784937 (+ 353us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 06:26:59.784944 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 06:26:59.785184 (+ 240us) client_negotiation.cc:769] Sending connection context
0806 06:26:59.785336 (+ 152us) client_negotiation.cc:240] Negotiation successful
0806 06:26:59.785617 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":101,"threads_started":1}
I20260806 06:26:59.786808 1514 heartbeater.cc:347] Connected to a master server at 127.21.12.190:46519
I20260806 06:26:59.786970 1514 heartbeater.cc:464] Registering TS with master...
I20260806 06:26:59.787232 1514 heartbeater.cc:511] Master 127.21.12.190:46519 requested a full tablet report, sending...
I20260806 06:26:59.787847 1004 ts_manager.cc:194] Registered new tserver with Master: 55552f6bc0d64fceb9de0589f1379083 (127.21.12.130:39719)
I20260806 06:26:59.788371 1004 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.12.130@KRBTEST.COM'} at 127.21.12.130:57663
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.12.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.12.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.131.keytab.
Entry for principal kudu/127.21.12.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.131.keytab.
I20260806 06:26:59.811038 21554 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHhLhT/build/release/bin/kudu
/tmp/dist-test-taskFHhLhT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785997581049705-21554-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785997581049705-21554-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785997581049705-21554-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785997581049705-21554-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.12.131:0
--local_ip_for_outbound_sockets=127.21.12.131
--webserver_interface=127.21.12.131
--webserver_port=0
--tserver_master_addrs=127.21.12.190:46519
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.131.keytab
--principal=kudu/127.21.12.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.148:37101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785997581049705-21554-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785997581049705-21554-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785997581049705-21554-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 06:26:59.955003 1526 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 06:26:59.955374 1526 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 06:26:59.955408 1526 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 06:26:59.957417 1526 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 06:26:59.957479 1526 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 06:26:59.957527 1526 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.12.131
I20260806 06:26:59.960016 1526 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.148:37101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785997581049705-21554-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785997581049705-21554-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.131.keytab
--principal=kudu/127.21.12.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.12.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785997581049705-21554-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.12.131
--webserver_port=0
--tserver_master_addrs=127.21.12.190:46519
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1526
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.12.131
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785997581049705-21554-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 05:11:02 UTC on ccycloud.quasar-hvzbbz.root.comops.site
build id 4569617
I20260806 06:26:59.960374 1526 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 06:26:59.960772 1526 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 06:26:59.968799 1535 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:26:59.968791 1532 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:26:59.968809 1533 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:26:59.969712 1526 server_base.cc:1034] running on GCE node
I20260806 06:26:59.970064 1526 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 06:26:59.970533 1526 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 06:26:59.971745 1526 hybrid_clock.cc:648] HybridClock initialized: now 1785997619971728 us; error 36 us; skew 500 ppm
Aug 06 06:26:59 dist-test-slave-4tdj krb5kdc[931](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997619, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 06:26:59.978626 1526 init.cc:377] Logged in from keytab as kudu/127.21.12.131@KRBTEST.COM (short username kudu)
I20260806 06:26:59.979389 1526 webserver.cc:460] Webserver started at http://127.21.12.131:40809/ using document root <none> and password file <none>
I20260806 06:26:59.979781 1526 fs_manager.cc:359] Metadata directory not provided
I20260806 06:26:59.979887 1526 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 06:26:59.980134 1526 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 06:26:59.981860 1526 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785997581049705-21554-0/minicluster-data/ts-2/data/instance:
uuid: "9b9fcfc41629465ebc00b4f29d0099f7"
format_stamp: "Formatted at 2026-08-06 06:26:59 on dist-test-slave-4tdj"
server_key: "088b9b0b452f57cbdf4fdec368c662fc"
server_key_iv: "4ed04ca0e08db6583c212b2c12921c79"
server_key_version: "encryptionkey@0"
I20260806 06:26:59.982575 1526 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785997581049705-21554-0/minicluster-data/ts-2/wal/instance:
uuid: "9b9fcfc41629465ebc00b4f29d0099f7"
format_stamp: "Formatted at 2026-08-06 06:26:59 on dist-test-slave-4tdj"
server_key: "088b9b0b452f57cbdf4fdec368c662fc"
server_key_iv: "4ed04ca0e08db6583c212b2c12921c79"
server_key_version: "encryptionkey@0"
I20260806 06:26:59.984961 1526 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.001s
I20260806 06:26:59.986230 1544 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 06:26:59.986534 1526 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 06:26:59.986738 1526 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785997581049705-21554-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785997581049705-21554-0/minicluster-data/ts-2/wal
uuid: "9b9fcfc41629465ebc00b4f29d0099f7"
format_stamp: "Formatted at 2026-08-06 06:26:59 on dist-test-slave-4tdj"
server_key: "088b9b0b452f57cbdf4fdec368c662fc"
server_key_iv: "4ed04ca0e08db6583c212b2c12921c79"
server_key_version: "encryptionkey@0"
I20260806 06:26:59.986881 1526 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785997581049705-21554-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785997581049705-21554-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785997581049705-21554-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 06:27:00.013098 1526 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 06:27:00.017164 1526 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 06:27:00.017434 1526 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 06:27:00.017932 1526 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 06:27:00.018563 1526 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 06:27:00.018613 1526 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 06:27:00.018659 1526 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 06:27:00.018700 1526 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 06:27:00.029681 1526 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.131:35773
I20260806 06:27:00.029798 1717 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.131:35773 every 8 connection(s)
I20260806 06:27:00.030445 1526 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785997581049705-21554-0/minicluster-data/ts-2/data/info.pb
I20260806 06:27:00.040307 21554 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHhLhT/build/release/bin/kudu as pid 1526
I20260806 06:27:00.040537 21554 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785997581049705-21554-0/minicluster-data/ts-2/wal/instance
I20260806 06:27:00.040937 21554 external_mini_cluster.cc:1428] Setting key 22a1b1216f057de1f565f4e942ec48d6
Aug 06 06:27:00 dist-test-slave-4tdj krb5kdc[931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997619, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.131@KRBTEST.COM for kudu/127.21.12.190@KRBTEST.COM
I20260806 06:27:00.047228 1066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:27:00.031597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.131:33059 (local address 127.21.12.190:46519)
0806 06:27:00.031671 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0806 06:27:00.031684 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 06:27:00.032122 (+ 438us) server_negotiation.cc:407] Connection header received
0806 06:27:00.032946 (+ 824us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:27:00.032948 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:27:00.032961 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:27:00.033075 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:27:00.034516 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:27:00.035324 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:27:00.037339 (+ 2015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:27:00.037880 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:27:00.038022 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:27:00.042323 (+ 4301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:27:00.042329 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:27:00.042335 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 06:27:00.042364 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:27:00.044701 (+ 2337us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:27:00.045836 (+ 1135us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:27:00.045837 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:27:00.045837 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:27:00.045929 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:27:00.046338 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:27:00.046339 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:27:00.046339 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:27:00.046515 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:27:00.046607 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0806 06:27:00.047023 (+ 416us) server_negotiation.cc:299] Negotiation successful
0806 06:27:00.047131 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260806 06:27:00.047693 1722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:27:00.031575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:46519 (local address 127.21.12.131:33059)
0806 06:27:00.031831 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0806 06:27:00.031928 (+ 97us) client_negotiation.cc:174] Beginning negotiation
0806 06:27:00.032800 (+ 872us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:27:00.033197 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:27:00.033201 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:27:00.034135 (+ 934us) client_negotiation.cc:189] Negotiated authn=SASL
0806 06:27:00.034430 (+ 295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:27:00.034437 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:27:00.035424 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:27:00.035425 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:27:00.036688 (+ 1263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:27:00.036690 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:27:00.038045 (+ 1355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:27:00.038046 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:27:00.038275 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:27:00.039278 (+ 1003us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 06:27:00.039296 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 06:27:00.042085 (+ 2789us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 06:27:00.044813 (+ 2728us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:27:00.044830 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:27:00.044832 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:27:00.045717 (+ 885us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:27:00.046012 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:27:00.046013 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:27:00.046013 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:27:00.046218 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:27:00.046627 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 06:27:00.046635 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 06:27:00.046852 (+ 217us) client_negotiation.cc:769] Sending connection context
0806 06:27:00.047004 (+ 152us) client_negotiation.cc:240] Negotiation successful
0806 06:27:00.047213 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":125,"threads_started":1}
I20260806 06:27:00.048595 1720 heartbeater.cc:347] Connected to a master server at 127.21.12.190:46519
I20260806 06:27:00.048750 1720 heartbeater.cc:464] Registering TS with master...
I20260806 06:27:00.049019 1720 heartbeater.cc:511] Master 127.21.12.190:46519 requested a full tablet report, sending...
I20260806 06:27:00.049584 1004 ts_manager.cc:194] Registered new tserver with Master: 9b9fcfc41629465ebc00b4f29d0099f7 (127.21.12.131:35773)
I20260806 06:27:00.050132 1004 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.12.131@KRBTEST.COM'} at 127.21.12.131:33059
I20260806 06:27:00.053776 21554 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.21.12.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.12.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.12.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785997581049705-21554-0/krb5kdc/oryx_127.21.12.129.keytab.
Entry for principal oryx/127.21.12.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785997581049705-21554-0/krb5kdc/oryx_127.21.12.129.keytab.
WARNING: no policy specified for oryx/127.21.12.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.12.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.12.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785997581049705-21554-0/krb5kdc/oryx_127.21.12.130.keytab.
Entry for principal oryx/127.21.12.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785997581049705-21554-0/krb5kdc/oryx_127.21.12.130.keytab.
WARNING: no policy specified for oryx/127.21.12.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.21.12.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.21.12.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785997581049705-21554-0/krb5kdc/oryx_127.21.12.131.keytab.
Entry for principal oryx/127.21.12.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785997581049705-21554-0/krb5kdc/oryx_127.21.12.131.keytab.
I20260806 06:27:00.156919 21554 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHhLhT/build/release/bin/kudu with pid 1104
I20260806 06:27:00.173380 21554 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHhLhT/build/release/bin/kudu with pid 1308
I20260806 06:27:00.191365 21554 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHhLhT/build/release/bin/kudu with pid 1526
I20260806 06:27:00.202936 21554 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHhLhT/build/release/bin/kudu with pid 955
I20260806 06:27:00.217468 21554 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHhLhT/build/release/bin/kudu
/tmp/dist-test-taskFHhLhT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785997581049705-21554-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785997581049705-21554-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785997581049705-21554-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785997581049705-21554-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.12.190:46519
--webserver_interface=127.21.12.190
--webserver_port=35413
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.190.keytab
--principal=kudu/127.21.12.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.148:37101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.12.190:46519
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785997581049705-21554-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785997581049705-21554-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785997581049705-21554-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 06:27:00.359448 1730 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 06:27:00.359851 1730 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 06:27:00.359937 1730 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 06:27:00.361815 1730 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 06:27:00.361907 1730 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 06:27:00.361958 1730 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 06:27:00.362013 1730 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 06:27:00.362056 1730 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 06:27:00.364595 1730 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.148:37101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785997581049705-21554-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785997581049705-21554-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.12.190:46519
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.190.keytab
--principal=kudu/127.21.12.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.12.190:46519
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785997581049705-21554-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.12.190
--webserver_port=35413
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1730
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785997581049705-21554-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 05:11:02 UTC on ccycloud.quasar-hvzbbz.root.comops.site
build id 4569617
I20260806 06:27:00.365049 1730 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 06:27:00.365432 1730 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 06:27:00.373377 1737 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:27:00.373442 1741 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:27:00.373793 1730 server_base.cc:1034] running on GCE node
W20260806 06:27:00.373473 1739 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:27:00.374244 1730 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 06:27:00.374785 1730 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 06:27:00.375965 1730 hybrid_clock.cc:648] HybridClock initialized: now 1785997620375946 us; error 38 us; skew 500 ppm
Aug 06 06:27:00 dist-test-slave-4tdj krb5kdc[931](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997620, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 06:27:00.380235 1730 init.cc:377] Logged in from keytab as kudu/127.21.12.190@KRBTEST.COM (short username kudu)
I20260806 06:27:00.380831 1730 webserver.cc:460] Webserver started at http://127.21.12.190:35413/ using document root <none> and password file <none>
I20260806 06:27:00.381170 1730 fs_manager.cc:359] Metadata directory not provided
I20260806 06:27:00.381311 1730 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 06:27:00.383852 1730 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260806 06:27:00.384990 1752 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 06:27:00.385277 1730 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260806 06:27:00.385462 1730 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785997581049705-21554-0/minicluster-data/master-0/data,/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785997581049705-21554-0/minicluster-data/master-0/wal
uuid: "980b938cd96e4402bf5882896a7bc3eb"
format_stamp: "Formatted at 2026-08-06 06:26:59 on dist-test-slave-4tdj"
server_key: "e1d4da835657ccf9d8d7cf0130b5564e"
server_key_iv: "d32f2e2ca24679ea7013801fc6e1be40"
server_key_version: "encryptionkey@0"
I20260806 06:27:00.386051 1730 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785997581049705-21554-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785997581049705-21554-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785997581049705-21554-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 06:27:00.403964 1730 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 06:27:00.405956 1730 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 06:27:00.406092 1730 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 06:27:00.409988 1730 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.190:46519
I20260806 06:27:00.410032 1827 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.190:46519 every 8 connection(s)
I20260806 06:27:00.410399 1730 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785997581049705-21554-0/minicluster-data/master-0/data/info.pb
I20260806 06:27:00.411821 1828 sys_catalog.cc:263] Verifying existing consensus state
I20260806 06:27:00.412467 1828 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 980b938cd96e4402bf5882896a7bc3eb: Bootstrap starting.
I20260806 06:27:00.415764 21554 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHhLhT/build/release/bin/kudu as pid 1730
I20260806 06:27:00.416029 21554 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHhLhT/build/release/bin/kudu
/tmp/dist-test-taskFHhLhT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785997581049705-21554-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785997581049705-21554-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785997581049705-21554-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785997581049705-21554-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.12.129:44089
--local_ip_for_outbound_sockets=127.21.12.129
--tserver_master_addrs=127.21.12.190:46519
--webserver_port=33681
--webserver_interface=127.21.12.129
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.129.keytab
--principal=kudu/127.21.12.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.148:37101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.21.12.129
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785997581049705-21554-0/krb5kdc/oryx_127.21.12.129.keytab with env {KRB5CCNAME=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785997581049705-21554-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785997581049705-21554-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785997581049705-21554-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 06:27:00.420943 1828 log.cc:824] T 00000000000000000000000000000000 P 980b938cd96e4402bf5882896a7bc3eb: Log is configured to *not* fsync() on all Append() calls
I20260806 06:27:00.422555 1828 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 980b938cd96e4402bf5882896a7bc3eb: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 06:27:00.422796 1828 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 980b938cd96e4402bf5882896a7bc3eb: Bootstrap complete.
I20260806 06:27:00.424865 1828 raft_consensus.cc:348] T 00000000000000000000000000000000 P 980b938cd96e4402bf5882896a7bc3eb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "980b938cd96e4402bf5882896a7bc3eb" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 46519 } }
I20260806 06:27:00.425086 1828 raft_consensus.cc:729] T 00000000000000000000000000000000 P 980b938cd96e4402bf5882896a7bc3eb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 980b938cd96e4402bf5882896a7bc3eb, State: Initialized, Role: FOLLOWER
I20260806 06:27:00.425259 1828 consensus_queue.cc:260] T 00000000000000000000000000000000 P 980b938cd96e4402bf5882896a7bc3eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "980b938cd96e4402bf5882896a7bc3eb" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 46519 } }
I20260806 06:27:00.425333 1828 raft_consensus.cc:388] T 00000000000000000000000000000000 P 980b938cd96e4402bf5882896a7bc3eb [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 06:27:00.425383 1828 raft_consensus.cc:482] T 00000000000000000000000000000000 P 980b938cd96e4402bf5882896a7bc3eb [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 06:27:00.425429 1828 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 980b938cd96e4402bf5882896a7bc3eb [term 1 FOLLOWER]: Advancing to term 2
I20260806 06:27:00.426383 1828 raft_consensus.cc:504] T 00000000000000000000000000000000 P 980b938cd96e4402bf5882896a7bc3eb [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "980b938cd96e4402bf5882896a7bc3eb" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 46519 } }
I20260806 06:27:00.426474 1828 leader_election.cc:302] T 00000000000000000000000000000000 P 980b938cd96e4402bf5882896a7bc3eb [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 980b938cd96e4402bf5882896a7bc3eb; no voters:
I20260806 06:27:00.426630 1828 leader_election.cc:288] T 00000000000000000000000000000000 P 980b938cd96e4402bf5882896a7bc3eb [CANDIDATE]: Term 2 election: Requested vote from peers
I20260806 06:27:00.426674 1833 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 980b938cd96e4402bf5882896a7bc3eb [term 2 FOLLOWER]: Leader election won for term 2
I20260806 06:27:00.426798 1833 raft_consensus.cc:686] T 00000000000000000000000000000000 P 980b938cd96e4402bf5882896a7bc3eb [term 2 LEADER]: Becoming Leader. State: Replica: 980b938cd96e4402bf5882896a7bc3eb, State: Running, Role: LEADER
I20260806 06:27:00.426890 1828 sys_catalog.cc:564] T 00000000000000000000000000000000 P 980b938cd96e4402bf5882896a7bc3eb [sys.catalog]: configured and running, proceeding with master startup.
I20260806 06:27:00.426882 1833 consensus_queue.cc:237] T 00000000000000000000000000000000 P 980b938cd96e4402bf5882896a7bc3eb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "980b938cd96e4402bf5882896a7bc3eb" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 46519 } }
I20260806 06:27:00.427214 1837 sys_catalog.cc:455] T 00000000000000000000000000000000 P 980b938cd96e4402bf5882896a7bc3eb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "980b938cd96e4402bf5882896a7bc3eb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "980b938cd96e4402bf5882896a7bc3eb" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 46519 } } }
I20260806 06:27:00.427309 1837 sys_catalog.cc:458] T 00000000000000000000000000000000 P 980b938cd96e4402bf5882896a7bc3eb [sys.catalog]: This master's current role is: LEADER
I20260806 06:27:00.427549 1838 sys_catalog.cc:455] T 00000000000000000000000000000000 P 980b938cd96e4402bf5882896a7bc3eb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 980b938cd96e4402bf5882896a7bc3eb. Latest consensus state: current_term: 2 leader_uuid: "980b938cd96e4402bf5882896a7bc3eb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "980b938cd96e4402bf5882896a7bc3eb" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 46519 } } }
I20260806 06:27:00.427623 1844 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 06:27:00.427661 1838 sys_catalog.cc:458] T 00000000000000000000000000000000 P 980b938cd96e4402bf5882896a7bc3eb [sys.catalog]: This master's current role is: LEADER
I20260806 06:27:00.428120 1844 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 06:27:00.428270 1844 catalog_manager.cc:1259] Loaded cluster ID: a0474fe8137641c28a8fbe8035e05786
I20260806 06:27:00.428462 1844 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 06:27:00.429251 1844 catalog_manager.cc:1504] Loading token signing keys...
I20260806 06:27:00.429404 1844 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 980b938cd96e4402bf5882896a7bc3eb: Loaded TSK: 0
I20260806 06:27:00.429598 1844 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260806 06:27:00.513935 1830 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 06:27:00.514145 1830 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 06:27:00.514170 1830 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 06:27:00.515329 1830 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 06:27:00.515376 1830 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 06:27:00.515404 1830 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.12.129
I20260806 06:27:00.516819 1830 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.148:37101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785997581049705-21554-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785997581049705-21554-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785997581049705-21554-0/krb5kdc/oryx_127.21.12.129.keytab
--principal=oryx/127.21.12.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.12.129:44089
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785997581049705-21554-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.12.129
--webserver_port=33681
--tserver_master_addrs=127.21.12.190:46519
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1830
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.12.129
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785997581049705-21554-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 05:11:02 UTC on ccycloud.quasar-hvzbbz.root.comops.site
build id 4569617
I20260806 06:27:00.517019 1830 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 06:27:00.517237 1830 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 06:27:00.523088 1871 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:27:00.523211 1830 server_base.cc:1034] running on GCE node
W20260806 06:27:00.523095 1874 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:27:00.523147 1870 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:27:00.523567 1830 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 06:27:00.523801 1830 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 06:27:00.524952 1830 hybrid_clock.cc:648] HybridClock initialized: now 1785997620524940 us; error 30 us; skew 500 ppm
Aug 06 06:27:00 dist-test-slave-4tdj krb5kdc[931](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997620, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.12.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 06:27:00.527606 1830 init.cc:377] Logged in from keytab as oryx/127.21.12.129@KRBTEST.COM (short username oryx)
I20260806 06:27:00.528075 1830 webserver.cc:460] Webserver started at http://127.21.12.129:33681/ using document root <none> and password file <none>
I20260806 06:27:00.528286 1830 fs_manager.cc:359] Metadata directory not provided
I20260806 06:27:00.528352 1830 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 06:27:00.529731 1830 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260806 06:27:00.530345 1883 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 06:27:00.530531 1830 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260806 06:27:00.530640 1830 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785997581049705-21554-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785997581049705-21554-0/minicluster-data/ts-0/wal
uuid: "8d4e9220c4c544d9b3804aa46a8da5e2"
format_stamp: "Formatted at 2026-08-06 06:26:59 on dist-test-slave-4tdj"
server_key: "a62215c8febf74b08f8705ed1186a69d"
server_key_iv: "92202797ec66aa228bafee9615ac93af"
server_key_version: "encryptionkey@0"
I20260806 06:27:00.531059 1830 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785997581049705-21554-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785997581049705-21554-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785997581049705-21554-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 06:27:00.555604 1830 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 06:27:00.558131 1830 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 06:27:00.558318 1830 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 06:27:00.558611 1830 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 06:27:00.558997 1830 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 06:27:00.559051 1830 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 06:27:00.559083 1830 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 06:27:00.559108 1830 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 06:27:00.566131 1830 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.129:44089
I20260806 06:27:00.566203 2063 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.129:44089 every 8 connection(s)
I20260806 06:27:00.566603 1830 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785997581049705-21554-0/minicluster-data/ts-0/data/info.pb
I20260806 06:27:00.572105 21554 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHhLhT/build/release/bin/kudu as pid 1830
I20260806 06:27:00.572422 21554 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHhLhT/build/release/bin/kudu
/tmp/dist-test-taskFHhLhT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785997581049705-21554-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785997581049705-21554-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785997581049705-21554-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785997581049705-21554-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.12.130:39719
--local_ip_for_outbound_sockets=127.21.12.130
--tserver_master_addrs=127.21.12.190:46519
--webserver_port=40207
--webserver_interface=127.21.12.130
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.130.keytab
--principal=kudu/127.21.12.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.148:37101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.21.12.130
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785997581049705-21554-0/krb5kdc/oryx_127.21.12.130.keytab with env {KRB5CCNAME=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785997581049705-21554-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785997581049705-21554-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785997581049705-21554-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 06:27:00 dist-test-slave-4tdj krb5kdc[931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.12.129@KRBTEST.COM for oryx/127.21.12.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 06:27:00 dist-test-slave-4tdj krb5kdc[931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.12.129@KRBTEST.COM for oryx/127.21.12.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 06:27:00.575659 2068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:27:00.567448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:46519 (local address 127.21.12.129:40109)
0806 06:27:00.567811 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0806 06:27:00.567848 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0806 06:27:00.568319 (+ 471us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:27:00.568837 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:27:00.568841 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:27:00.569420 (+ 579us) client_negotiation.cc:189] Negotiated authn=SASL
0806 06:27:00.569613 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:27:00.569618 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:27:00.570761 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:27:00.570762 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:27:00.571533 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:27:00.571535 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:27:00.572215 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:27:00.572216 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:27:00.572367 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:27:00.572887 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 06:27:00.572897 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0806 06:27:00.575452 (+ 2555us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.12.190:46519: Server oryx/127.21.12.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":92,"threads_started":1}
W20260806 06:27:00.575685 2071 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 06:27:00.567494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.129:40109 (local address 127.21.12.190:46519)
0806 06:27:00.568484 (+ 990us) server_negotiation.cc:206] Beginning negotiation
0806 06:27:00.568492 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 06:27:00.568519 (+ 27us) server_negotiation.cc:407] Connection header received
0806 06:27:00.568621 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:27:00.568625 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:27:00.568663 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:27:00.568775 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:27:00.569730 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:27:00.570671 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:27:00.571620 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:27:00.572108 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:27:00.572299 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:27:00.575561 (+ 3262us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.12.129:40109: BlockingRecv error: recv got EOF from 127.21.12.129:40109 (error 108)
Metrics: {"server-negotiator.queue_time_us":938,"thread_start_us":131,"threads_started":1}
W20260806 06:27:00.575897 2066 heartbeater.cc:644] Failed to heartbeat to 127.21.12.190:46519 (0 consecutive failures): Not authorized: Failed to ping master at 127.21.12.190:46519: Client connection negotiation failed: client connection to 127.21.12.190:46519: Server oryx/127.21.12.190@KRBTEST.COM not found in Kerberos database
Aug 06 06:27:00 dist-test-slave-4tdj krb5kdc[931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.12.129@KRBTEST.COM for oryx/127.21.12.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 06:27:00 dist-test-slave-4tdj krb5kdc[931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.12.129@KRBTEST.COM for oryx/127.21.12.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 06:27:00.580813 2068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:27:00.576125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:46519 (local address 127.21.12.129:33067)
0806 06:27:00.576169 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0806 06:27:00.576178 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0806 06:27:00.576213 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:27:00.576425 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:27:00.576426 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:27:00.576542 (+ 116us) client_negotiation.cc:189] Negotiated authn=SASL
0806 06:27:00.576587 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:27:00.576588 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:27:00.577173 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:27:00.577174 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:27:00.577825 (+ 651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:27:00.577829 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:27:00.578330 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:27:00.578331 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:27:00.578395 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:27:00.578844 (+ 449us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 06:27:00.578850 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0806 06:27:00.580743 (+ 1893us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.12.190:46519: Server oryx/127.21.12.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":29}
W20260806 06:27:00.580896 2073 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 06:27:00.576180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.129:33067 (local address 127.21.12.190:46519)
0806 06:27:00.576313 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0806 06:27:00.576314 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0806 06:27:00.576321 (+ 7us) server_negotiation.cc:407] Connection header received
0806 06:27:00.576341 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:27:00.576341 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:27:00.576350 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:27:00.576410 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:27:00.576651 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:27:00.577109 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:27:00.577927 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:27:00.578240 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:27:00.578349 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:27:00.580837 (+ 2488us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.12.129:33067: BlockingRecv error: recv got EOF from 127.21.12.129:33067 (error 108)
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":81,"threads_started":1}
Aug 06 06:27:00 dist-test-slave-4tdj krb5kdc[931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.12.129@KRBTEST.COM for oryx/127.21.12.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 06:27:00 dist-test-slave-4tdj krb5kdc[931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.12.129@KRBTEST.COM for oryx/127.21.12.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 06:27:00.585495 2068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:27:00.581281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:46519 (local address 127.21.12.129:40395)
0806 06:27:00.581324 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0806 06:27:00.581333 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0806 06:27:00.581363 (+ 30us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:27:00.581488 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:27:00.581489 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:27:00.581622 (+ 133us) client_negotiation.cc:189] Negotiated authn=SASL
0806 06:27:00.581652 (+ 30us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:27:00.581653 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:27:00.582531 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:27:00.582532 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:27:00.583070 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:27:00.583073 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:27:00.583491 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:27:00.583492 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:27:00.583552 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:27:00.583928 (+ 376us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 06:27:00.583932 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0806 06:27:00.585428 (+ 1496us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.12.190:46519: Server oryx/127.21.12.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":30}
W20260806 06:27:00.585551 2073 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 06:27:00.581341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.129:40395 (local address 127.21.12.190:46519)
0806 06:27:00.581376 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0806 06:27:00.581378 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:27:00.581385 (+ 7us) server_negotiation.cc:407] Connection header received
0806 06:27:00.581407 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:27:00.581407 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:27:00.581417 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:27:00.581472 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:27:00.581719 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:27:00.582429 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:27:00.583147 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:27:00.583420 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:27:00.583489 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:27:00.585502 (+ 2013us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.12.129:40395: BlockingRecv error: recv got EOF from 127.21.12.129:40395 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260806 06:27:00.585717 2066 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260806 06:27:00.656301 2072 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 06:27:00.656489 2072 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 06:27:00.656510 2072 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 06:27:00.657686 2072 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 06:27:00.657727 2072 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 06:27:00.657752 2072 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.12.130
I20260806 06:27:00.659170 2072 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.148:37101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785997581049705-21554-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785997581049705-21554-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785997581049705-21554-0/krb5kdc/oryx_127.21.12.130.keytab
--principal=oryx/127.21.12.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.12.130:39719
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785997581049705-21554-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.12.130
--webserver_port=40207
--tserver_master_addrs=127.21.12.190:46519
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2072
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.12.130
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785997581049705-21554-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 05:11:02 UTC on ccycloud.quasar-hvzbbz.root.comops.site
build id 4569617
I20260806 06:27:00.659372 2072 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 06:27:00.659572 2072 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 06:27:00.664254 2087 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:27:00.664276 2085 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:27:00.664285 2084 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:27:00.664580 2072 server_base.cc:1034] running on GCE node
I20260806 06:27:00.664785 2072 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 06:27:00.664989 2072 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 06:27:00.666124 2072 hybrid_clock.cc:648] HybridClock initialized: now 1785997620666123 us; error 35 us; skew 500 ppm
Aug 06 06:27:00 dist-test-slave-4tdj krb5kdc[931](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997620, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.12.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 06:27:00.668462 2072 init.cc:377] Logged in from keytab as oryx/127.21.12.130@KRBTEST.COM (short username oryx)
I20260806 06:27:00.668829 2072 webserver.cc:460] Webserver started at http://127.21.12.130:40207/ using document root <none> and password file <none>
I20260806 06:27:00.668995 2072 fs_manager.cc:359] Metadata directory not provided
I20260806 06:27:00.669055 2072 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 06:27:00.670215 2072 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260806 06:27:00.670785 2098 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 06:27:00.670928 2072 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260806 06:27:00.671008 2072 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785997581049705-21554-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785997581049705-21554-0/minicluster-data/ts-1/wal
uuid: "55552f6bc0d64fceb9de0589f1379083"
format_stamp: "Formatted at 2026-08-06 06:26:59 on dist-test-slave-4tdj"
server_key: "bf2d09065f906035b9b65fd2da5a2c25"
server_key_iv: "05bc261fec486115c38e31e54fa460d6"
server_key_version: "encryptionkey@0"
I20260806 06:27:00.671320 2072 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785997581049705-21554-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785997581049705-21554-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785997581049705-21554-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 06:27:00.693917 2072 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 06:27:00.695729 2072 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 06:27:00.695868 2072 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 06:27:00.696094 2072 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 06:27:00.696389 2072 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 06:27:00.696424 2072 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 06:27:00.696453 2072 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 06:27:00.696475 2072 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 06:27:00.702040 2072 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.130:39719
I20260806 06:27:00.702092 2270 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.130:39719 every 8 connection(s)
I20260806 06:27:00.702438 2072 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785997581049705-21554-0/minicluster-data/ts-1/data/info.pb
Aug 06 06:27:00 dist-test-slave-4tdj krb5kdc[931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.12.130@KRBTEST.COM for oryx/127.21.12.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 06:27:00 dist-test-slave-4tdj krb5kdc[931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.12.130@KRBTEST.COM for oryx/127.21.12.190@KRBTEST.COM, Server not found in Kerberos database
W20260806 06:27:00.709228 2073 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 06:27:00.703097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.130:40767 (local address 127.21.12.190:46519)
0806 06:27:00.703155 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0806 06:27:00.703156 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0806 06:27:00.703353 (+ 197us) server_negotiation.cc:407] Connection header received
0806 06:27:00.703853 (+ 500us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:27:00.703854 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:27:00.703870 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:27:00.703919 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:27:00.704853 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:27:00.705361 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:27:00.706330 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:27:00.706521 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:27:00.706578 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:27:00.709173 (+ 2595us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.12.130:40767: BlockingRecv error: recv got EOF from 127.21.12.130:40767 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
I20260806 06:27:00.709273 2275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:27:00.703084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:46519 (local address 127.21.12.130:40767)
0806 06:27:00.703237 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0806 06:27:00.703273 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0806 06:27:00.703757 (+ 484us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:27:00.704002 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:27:00.704006 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:27:00.704569 (+ 563us) client_negotiation.cc:189] Negotiated authn=SASL
0806 06:27:00.704759 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:27:00.704763 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:27:00.705443 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:27:00.705444 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:27:00.706228 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:27:00.706231 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:27:00.706592 (+ 361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:27:00.706594 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:27:00.706716 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:27:00.707137 (+ 421us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 06:27:00.707145 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0806 06:27:00.709086 (+ 1941us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.12.190:46519: Server oryx/127.21.12.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":69,"threads_started":1}
W20260806 06:27:00.709450 2273 heartbeater.cc:644] Failed to heartbeat to 127.21.12.190:46519 (0 consecutive failures): Not authorized: Failed to ping master at 127.21.12.190:46519: Client connection negotiation failed: client connection to 127.21.12.190:46519: Server oryx/127.21.12.190@KRBTEST.COM not found in Kerberos database
I20260806 06:27:00.710515 21554 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHhLhT/build/release/bin/kudu as pid 2072
I20260806 06:27:00.710817 21554 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHhLhT/build/release/bin/kudu
/tmp/dist-test-taskFHhLhT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785997581049705-21554-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785997581049705-21554-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785997581049705-21554-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785997581049705-21554-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.12.131:35773
--local_ip_for_outbound_sockets=127.21.12.131
--tserver_master_addrs=127.21.12.190:46519
--webserver_port=40809
--webserver_interface=127.21.12.131
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.131.keytab
--principal=kudu/127.21.12.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.148:37101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.21.12.131
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785997581049705-21554-0/krb5kdc/oryx_127.21.12.131.keytab with env {KRB5CCNAME=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785997581049705-21554-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785997581049705-21554-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785997581049705-21554-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 06:27:00 dist-test-slave-4tdj krb5kdc[931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.12.130@KRBTEST.COM for oryx/127.21.12.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 06:27:00 dist-test-slave-4tdj krb5kdc[931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.12.130@KRBTEST.COM for oryx/127.21.12.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 06:27:00.713707 2275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:27:00.709645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:46519 (local address 127.21.12.130:39437)
0806 06:27:00.709698 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0806 06:27:00.709707 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0806 06:27:00.709752 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:27:00.709874 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:27:00.709875 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:27:00.709993 (+ 118us) client_negotiation.cc:189] Negotiated authn=SASL
0806 06:27:00.710033 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:27:00.710034 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:27:00.710792 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:27:00.710793 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:27:00.711317 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:27:00.711318 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:27:00.711713 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:27:00.711714 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:27:00.711761 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:27:00.712104 (+ 343us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 06:27:00.712108 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0806 06:27:00.713654 (+ 1546us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.12.190:46519: Server oryx/127.21.12.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":36}
W20260806 06:27:00.713783 2073 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 06:27:00.709698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.130:39437 (local address 127.21.12.190:46519)
0806 06:27:00.709748 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0806 06:27:00.709749 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0806 06:27:00.709757 (+ 8us) server_negotiation.cc:407] Connection header received
0806 06:27:00.709803 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:27:00.709804 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:27:00.709816 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:27:00.709864 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:27:00.710095 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:27:00.710723 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:27:00.711390 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:27:00.711649 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:27:00.711716 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:27:00.713727 (+ 2011us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.12.130:39437: BlockingRecv error: recv got EOF from 127.21.12.130:39437 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
Aug 06 06:27:00 dist-test-slave-4tdj krb5kdc[931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.12.130@KRBTEST.COM for oryx/127.21.12.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 06:27:00 dist-test-slave-4tdj krb5kdc[931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.12.130@KRBTEST.COM for oryx/127.21.12.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 06:27:00.717764 2275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:27:00.713929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:46519 (local address 127.21.12.130:37013)
0806 06:27:00.713961 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0806 06:27:00.713969 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0806 06:27:00.713998 (+ 29us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:27:00.714120 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:27:00.714121 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:27:00.714243 (+ 122us) client_negotiation.cc:189] Negotiated authn=SASL
0806 06:27:00.714273 (+ 30us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:27:00.714275 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:27:00.714846 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:27:00.714848 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:27:00.715565 (+ 717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:27:00.715568 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:27:00.715866 (+ 298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:27:00.715867 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:27:00.715915 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:27:00.716223 (+ 308us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 06:27:00.716228 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0806 06:27:00.717709 (+ 1481us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.12.190:46519: Server oryx/127.21.12.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":20}
W20260806 06:27:00.717845 2073 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 06:27:00.713973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.130:37013 (local address 127.21.12.190:46519)
0806 06:27:00.714017 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0806 06:27:00.714019 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:27:00.714026 (+ 7us) server_negotiation.cc:407] Connection header received
0806 06:27:00.714046 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:27:00.714047 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:27:00.714054 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:27:00.714104 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:27:00.714347 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:27:00.714773 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:27:00.715637 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:27:00.715805 (+ 168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:27:00.715852 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:27:00.717796 (+ 1944us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.12.130:37013: BlockingRecv error: recv got EOF from 127.21.12.130:37013 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
W20260806 06:27:00.717882 2273 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260806 06:27:00.791965 2276 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 06:27:00.792204 2276 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 06:27:00.792240 2276 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 06:27:00.793396 2276 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 06:27:00.793437 2276 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 06:27:00.793473 2276 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.12.131
I20260806 06:27:00.794960 2276 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.148:37101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785997581049705-21554-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785997581049705-21554-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785997581049705-21554-0/krb5kdc/oryx_127.21.12.131.keytab
--principal=oryx/127.21.12.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.12.131:35773
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785997581049705-21554-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.12.131
--webserver_port=40809
--tserver_master_addrs=127.21.12.190:46519
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2276
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.12.131
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785997581049705-21554-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 05:11:02 UTC on ccycloud.quasar-hvzbbz.root.comops.site
build id 4569617
I20260806 06:27:00.795210 2276 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 06:27:00.795424 2276 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 06:27:00.800354 2288 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:27:00.800328 2281 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:27:00.800333 2276 server_base.cc:1034] running on GCE node
W20260806 06:27:00.800376 2282 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:27:00.800716 2276 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 06:27:00.800941 2276 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 06:27:00.802091 2276 hybrid_clock.cc:648] HybridClock initialized: now 1785997620802084 us; error 35 us; skew 500 ppm
Aug 06 06:27:00 dist-test-slave-4tdj krb5kdc[931](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997620, etypes {rep=17 tkt=17 ses=17}, oryx/127.21.12.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 06:27:00.804497 2276 init.cc:377] Logged in from keytab as oryx/127.21.12.131@KRBTEST.COM (short username oryx)
I20260806 06:27:00.804919 2276 webserver.cc:460] Webserver started at http://127.21.12.131:40809/ using document root <none> and password file <none>
I20260806 06:27:00.805158 2276 fs_manager.cc:359] Metadata directory not provided
I20260806 06:27:00.805234 2276 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 06:27:00.806591 2276 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260806 06:27:00.807183 2300 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 06:27:00.807348 2276 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260806 06:27:00.807428 2276 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785997581049705-21554-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785997581049705-21554-0/minicluster-data/ts-2/wal
uuid: "9b9fcfc41629465ebc00b4f29d0099f7"
format_stamp: "Formatted at 2026-08-06 06:26:59 on dist-test-slave-4tdj"
server_key: "088b9b0b452f57cbdf4fdec368c662fc"
server_key_iv: "4ed04ca0e08db6583c212b2c12921c79"
server_key_version: "encryptionkey@0"
I20260806 06:27:00.807749 2276 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785997581049705-21554-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785997581049705-21554-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785997581049705-21554-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 06:27:00.825986 2276 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 06:27:00.827999 2276 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 06:27:00.828115 2276 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 06:27:00.828351 2276 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 06:27:00.828667 2276 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 06:27:00.828701 2276 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 06:27:00.828732 2276 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 06:27:00.828755 2276 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 06:27:00.834251 2276 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.131:35773
I20260806 06:27:00.834307 2477 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.131:35773 every 8 connection(s)
I20260806 06:27:00.834671 2276 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785997581049705-21554-0/minicluster-data/ts-2/data/info.pb
I20260806 06:27:00.836511 21554 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHhLhT/build/release/bin/kudu as pid 2276
Aug 06 06:27:00 dist-test-slave-4tdj krb5kdc[931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.12.131@KRBTEST.COM for oryx/127.21.12.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 06:27:00 dist-test-slave-4tdj krb5kdc[931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.12.131@KRBTEST.COM for oryx/127.21.12.190@KRBTEST.COM, Server not found in Kerberos database
W20260806 06:27:00.842283 2073 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 06:27:00.835560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.131:47517 (local address 127.21.12.190:46519)
0806 06:27:00.835617 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0806 06:27:00.835618 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0806 06:27:00.835853 (+ 235us) server_negotiation.cc:407] Connection header received
0806 06:27:00.836332 (+ 479us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:27:00.836334 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:27:00.836347 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:27:00.836422 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:27:00.837505 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:27:00.838084 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:27:00.839162 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:27:00.839357 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:27:00.839418 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:27:00.842235 (+ 2817us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.12.131:47517: BlockingRecv error: recv got EOF from 127.21.12.131:47517 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
I20260806 06:27:00.842401 2481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:27:00.835550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:46519 (local address 127.21.12.131:47517)
0806 06:27:00.835735 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0806 06:27:00.835770 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0806 06:27:00.836252 (+ 482us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:27:00.836480 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:27:00.836483 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:27:00.837208 (+ 725us) client_negotiation.cc:189] Negotiated authn=SASL
0806 06:27:00.837414 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:27:00.837419 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:27:00.838162 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:27:00.838163 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:27:00.839080 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:27:00.839083 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:27:00.839431 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:27:00.839432 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:27:00.839533 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:27:00.839981 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 06:27:00.839989 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0806 06:27:00.842176 (+ 2187us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.12.190:46519: Server oryx/127.21.12.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":111,"threads_started":1}
W20260806 06:27:00.842594 2478 heartbeater.cc:644] Failed to heartbeat to 127.21.12.190:46519 (0 consecutive failures): Not authorized: Failed to ping master at 127.21.12.190:46519: Client connection negotiation failed: client connection to 127.21.12.190:46519: Server oryx/127.21.12.190@KRBTEST.COM not found in Kerberos database
I20260806 06:27:00.843573 2071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:27:00.836973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38452 (local address 127.21.12.190:46519)
0806 06:27:00.837035 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0806 06:27:00.837037 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:27:00.837045 (+ 8us) server_negotiation.cc:407] Connection header received
0806 06:27:00.837076 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:27:00.837077 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:27:00.837089 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:27:00.837154 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:27:00.837749 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:27:00.838267 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:27:00.839009 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:27:00.839267 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:27:00.839315 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:27:00.840458 (+ 1143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:27:00.840463 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:27:00.840464 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 06:27:00.840476 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:27:00.842034 (+ 1558us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:27:00.842535 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:27:00.842538 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:27:00.842539 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:27:00.842655 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:27:00.842917 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:27:00.842918 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:27:00.842919 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:27:00.843145 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:27:00.843270 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0806 06:27:00.843386 (+ 116us) server_negotiation.cc:299] Negotiation successful
0806 06:27:00.843490 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
Aug 06 06:27:00 dist-test-slave-4tdj krb5kdc[931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.12.131@KRBTEST.COM for oryx/127.21.12.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 06:27:00 dist-test-slave-4tdj krb5kdc[931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.12.131@KRBTEST.COM for oryx/127.21.12.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 06:27:00.846659 2481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:27:00.842815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:46519 (local address 127.21.12.131:51281)
0806 06:27:00.842851 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0806 06:27:00.842861 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0806 06:27:00.842897 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:27:00.843010 (+ 113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:27:00.843010 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:27:00.843138 (+ 128us) client_negotiation.cc:189] Negotiated authn=SASL
0806 06:27:00.843176 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:27:00.843177 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:27:00.843845 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:27:00.843846 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:27:00.844458 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:27:00.844461 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:27:00.844745 (+ 284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:27:00.844746 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:27:00.844789 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:27:00.845136 (+ 347us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 06:27:00.845141 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0806 06:27:00.846619 (+ 1478us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.12.190:46519: Server oryx/127.21.12.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":19}
W20260806 06:27:00.846745 2073 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 06:27:00.842852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.131:51281 (local address 127.21.12.190:46519)
0806 06:27:00.842890 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0806 06:27:00.842892 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:27:00.842899 (+ 7us) server_negotiation.cc:407] Connection header received
0806 06:27:00.842949 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:27:00.842949 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:27:00.842957 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:27:00.843001 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:27:00.843241 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:27:00.843777 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:27:00.844524 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:27:00.844699 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:27:00.844743 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:27:00.846687 (+ 1944us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.12.131:51281: BlockingRecv error: recv got EOF from 127.21.12.131:51281 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
Aug 06 06:27:00 dist-test-slave-4tdj krb5kdc[931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.12.131@KRBTEST.COM for oryx/127.21.12.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 06:27:00 dist-test-slave-4tdj krb5kdc[931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.12.131@KRBTEST.COM for oryx/127.21.12.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 06:27:00.850088 2481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:27:00.846874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:46519 (local address 127.21.12.131:59381)
0806 06:27:00.846909 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0806 06:27:00.846918 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0806 06:27:00.846947 (+ 29us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:27:00.847031 (+ 84us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:27:00.847032 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:27:00.847126 (+ 94us) client_negotiation.cc:189] Negotiated authn=SASL
0806 06:27:00.847148 (+ 22us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:27:00.847149 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:27:00.847654 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:27:00.847655 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:27:00.848102 (+ 447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:27:00.848103 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:27:00.848351 (+ 248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:27:00.848351 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:27:00.848382 (+ 31us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:27:00.848711 (+ 329us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 06:27:00.848716 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0806 06:27:00.850046 (+ 1330us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.12.190:46519: Server oryx/127.21.12.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":22}
W20260806 06:27:00.850172 2073 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 06:27:00.846915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.131:59381 (local address 127.21.12.190:46519)
0806 06:27:00.846946 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0806 06:27:00.846955 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 06:27:00.846962 (+ 7us) server_negotiation.cc:407] Connection header received
0806 06:27:00.846981 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:27:00.846982 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:27:00.846988 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:27:00.847029 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:27:00.847192 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:27:00.847591 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:27:00.848152 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:27:00.848309 (+ 157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:27:00.848347 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:27:00.850117 (+ 1770us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.12.131:59381: BlockingRecv error: recv got EOF from 127.21.12.131:59381 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
W20260806 06:27:00.850196 2478 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Aug 06 06:27:01 dist-test-slave-4tdj krb5kdc[931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.12.129@KRBTEST.COM for oryx/127.21.12.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 06:27:01 dist-test-slave-4tdj krb5kdc[931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.12.129@KRBTEST.COM for oryx/127.21.12.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 06:27:01.590322 2484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:27:01.586037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:46519 (local address 127.21.12.129:42427)
0806 06:27:01.586173 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0806 06:27:01.586187 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 06:27:01.586244 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:27:01.586393 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:27:01.586394 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:27:01.586552 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0806 06:27:01.586601 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:27:01.586602 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:27:01.587195 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:27:01.587196 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:27:01.587745 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:27:01.587747 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:27:01.588082 (+ 335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:27:01.588082 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:27:01.588140 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:27:01.588545 (+ 405us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 06:27:01.588550 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0806 06:27:01.590212 (+ 1662us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.12.190:46519: Server oryx/127.21.12.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":69,"threads_started":1}
W20260806 06:27:01.590394 2485 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 06:27:01.586094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.129:42427 (local address 127.21.12.190:46519)
0806 06:27:01.586222 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0806 06:27:01.586223 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0806 06:27:01.586231 (+ 8us) server_negotiation.cc:407] Connection header received
0806 06:27:01.586285 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:27:01.586286 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:27:01.586318 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:27:01.586383 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:27:01.586653 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:27:01.587140 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:27:01.587821 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:27:01.588021 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:27:01.588075 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:27:01.590334 (+ 2259us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.12.129:42427: BlockingRecv error: recv got EOF from 127.21.12.129:42427 (error 108)
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":65,"threads_started":1}
Aug 06 06:27:01 dist-test-slave-4tdj krb5kdc[931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.12.130@KRBTEST.COM for oryx/127.21.12.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 06:27:01 dist-test-slave-4tdj krb5kdc[931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.12.130@KRBTEST.COM for oryx/127.21.12.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 06:27:01.722468 2486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:27:01.718234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:46519 (local address 127.21.12.130:58689)
0806 06:27:01.718366 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0806 06:27:01.718377 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0806 06:27:01.718434 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:27:01.718549 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:27:01.718550 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:27:01.718710 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0806 06:27:01.718757 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:27:01.718759 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:27:01.719346 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:27:01.719347 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:27:01.719874 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:27:01.719875 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:27:01.720183 (+ 308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:27:01.720184 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:27:01.720228 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:27:01.720659 (+ 431us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 06:27:01.720664 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0806 06:27:01.722403 (+ 1739us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.12.190:46519: Server oryx/127.21.12.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":81,"threads_started":1}
W20260806 06:27:01.722533 2485 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 06:27:01.718320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.130:58689 (local address 127.21.12.190:46519)
0806 06:27:01.718364 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0806 06:27:01.718366 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:27:01.718428 (+ 62us) server_negotiation.cc:407] Connection header received
0806 06:27:01.718475 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:27:01.718476 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:27:01.718486 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:27:01.718535 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:27:01.718806 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:27:01.719290 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:27:01.719942 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:27:01.720123 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:27:01.720168 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:27:01.722484 (+ 2316us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.12.130:58689: BlockingRecv error: recv got EOF from 127.21.12.130:58689 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
Aug 06 06:27:01 dist-test-slave-4tdj krb5kdc[931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.12.131@KRBTEST.COM for oryx/127.21.12.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 06:27:01 dist-test-slave-4tdj krb5kdc[931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.12.131@KRBTEST.COM for oryx/127.21.12.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 06:27:01.855139 2488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:27:01.850561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:46519 (local address 127.21.12.131:50437)
0806 06:27:01.850696 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0806 06:27:01.850707 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0806 06:27:01.850771 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:27:01.850930 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:27:01.850932 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:27:01.851102 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0806 06:27:01.851148 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:27:01.851150 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:27:01.851897 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:27:01.851899 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:27:01.852467 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:27:01.852470 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:27:01.852831 (+ 361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:27:01.852832 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:27:01.852873 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:27:01.853269 (+ 396us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 06:27:01.853273 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0806 06:27:01.855040 (+ 1767us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.12.190:46519: Server oryx/127.21.12.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":65,"threads_started":1}
W20260806 06:27:01.855186 2485 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 06:27:01.850657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.131:50437 (local address 127.21.12.190:46519)
0806 06:27:01.850720 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0806 06:27:01.850721 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0806 06:27:01.850754 (+ 33us) server_negotiation.cc:407] Connection header received
0806 06:27:01.850821 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:27:01.850822 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:27:01.850834 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:27:01.850882 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:27:01.851231 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:27:01.851835 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:27:01.852577 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:27:01.852771 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:27:01.852826 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:27:01.855133 (+ 2307us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.12.131:50437: BlockingRecv error: recv got EOF from 127.21.12.131:50437 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Aug 06 06:27:02 dist-test-slave-4tdj krb5kdc[931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.12.129@KRBTEST.COM for oryx/127.21.12.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 06:27:02 dist-test-slave-4tdj krb5kdc[931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.12.129@KRBTEST.COM for oryx/127.21.12.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 06:27:02.594993 2489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:27:02.590770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:46519 (local address 127.21.12.129:45139)
0806 06:27:02.590879 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0806 06:27:02.590892 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 06:27:02.590953 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:27:02.591067 (+ 114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:27:02.591068 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:27:02.591230 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0806 06:27:02.591296 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:27:02.591298 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:27:02.591921 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:27:02.591922 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:27:02.592460 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:27:02.592462 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:27:02.592783 (+ 321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:27:02.592783 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:27:02.592819 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:27:02.593254 (+ 435us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 06:27:02.593259 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0806 06:27:02.594935 (+ 1676us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.12.190:46519: Server oryx/127.21.12.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":88,"thread_start_us":54,"threads_started":1}
W20260806 06:27:02.595083 2490 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 06:27:02.590819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.129:45139 (local address 127.21.12.190:46519)
0806 06:27:02.590947 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0806 06:27:02.590949 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:27:02.590957 (+ 8us) server_negotiation.cc:407] Connection header received
0806 06:27:02.590996 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:27:02.590997 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:27:02.591006 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:27:02.591063 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:27:02.591344 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:27:02.591863 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:27:02.592527 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:27:02.592729 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:27:02.592780 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:27:02.595016 (+ 2236us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.12.129:45139: BlockingRecv error: recv got EOF from 127.21.12.129:45139 (error 108)
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":70,"threads_started":1}
Aug 06 06:27:02 dist-test-slave-4tdj krb5kdc[931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.12.130@KRBTEST.COM for oryx/127.21.12.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 06:27:02 dist-test-slave-4tdj krb5kdc[931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.12.130@KRBTEST.COM for oryx/127.21.12.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 06:27:02.727283 2491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:27:02.722947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:46519 (local address 127.21.12.130:44183)
0806 06:27:02.723091 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0806 06:27:02.723101 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0806 06:27:02.723165 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:27:02.723286 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:27:02.723286 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:27:02.723440 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0806 06:27:02.723490 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:27:02.723492 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:27:02.724109 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:27:02.724111 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:27:02.724655 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:27:02.724657 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:27:02.724936 (+ 279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:27:02.724936 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:27:02.724971 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:27:02.725391 (+ 420us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 06:27:02.725395 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0806 06:27:02.727207 (+ 1812us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.12.190:46519: Server oryx/127.21.12.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":62,"threads_started":1}
W20260806 06:27:02.727356 2490 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 06:27:02.723010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.130:44183 (local address 127.21.12.190:46519)
0806 06:27:02.723058 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0806 06:27:02.723059 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0806 06:27:02.723152 (+ 93us) server_negotiation.cc:407] Connection header received
0806 06:27:02.723222 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:27:02.723223 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:27:02.723235 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:27:02.723273 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:27:02.723539 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:27:02.724037 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:27:02.724712 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:27:02.724888 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:27:02.724932 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:27:02.727307 (+ 2375us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.12.130:44183: BlockingRecv error: recv got EOF from 127.21.12.130:44183 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
Aug 06 06:27:02 dist-test-slave-4tdj krb5kdc[931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.12.131@KRBTEST.COM for oryx/127.21.12.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 06:27:02 dist-test-slave-4tdj krb5kdc[931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.12.131@KRBTEST.COM for oryx/127.21.12.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 06:27:02.860311 2492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:27:02.855599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:46519 (local address 127.21.12.131:38335)
0806 06:27:02.855744 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0806 06:27:02.855758 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 06:27:02.855825 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:27:02.855957 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:27:02.855958 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:27:02.856167 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0806 06:27:02.856215 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:27:02.856217 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:27:02.857056 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:27:02.857065 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:27:02.857596 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:27:02.857598 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:27:02.857932 (+ 334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:27:02.857934 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:27:02.857987 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:27:02.858475 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 06:27:02.858480 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0806 06:27:02.860252 (+ 1772us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.12.190:46519: Server oryx/127.21.12.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":60,"threads_started":1}
W20260806 06:27:02.860410 2490 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 06:27:02.855743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.131:38335 (local address 127.21.12.190:46519)
0806 06:27:02.855813 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0806 06:27:02.855815 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:27:02.855824 (+ 9us) server_negotiation.cc:407] Connection header received
0806 06:27:02.855879 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:27:02.855880 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:27:02.855894 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:27:02.855944 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:27:02.856293 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:27:02.856986 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:27:02.857677 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:27:02.857874 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:27:02.857929 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:27:02.860342 (+ 2413us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.12.131:38335: BlockingRecv error: recv got EOF from 127.21.12.131:38335 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Aug 06 06:27:03 dist-test-slave-4tdj krb5kdc[931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.12.129@KRBTEST.COM for oryx/127.21.12.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 06:27:03 dist-test-slave-4tdj krb5kdc[931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.12.129@KRBTEST.COM for oryx/127.21.12.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 06:27:03.599606 2494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:27:03.595417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:46519 (local address 127.21.12.129:51549)
0806 06:27:03.595539 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0806 06:27:03.595550 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0806 06:27:03.595596 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:27:03.595749 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:27:03.595750 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:27:03.595913 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0806 06:27:03.595962 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:27:03.595964 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:27:03.596561 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:27:03.596561 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:27:03.597084 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:27:03.597085 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:27:03.597353 (+ 268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:27:03.597353 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:27:03.597388 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:27:03.597824 (+ 436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 06:27:03.597829 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0806 06:27:03.599526 (+ 1697us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.12.190:46519: Server oryx/127.21.12.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":57,"threads_started":1}
W20260806 06:27:03.599704 2495 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 06:27:03.595477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.129:51549 (local address 127.21.12.190:46519)
0806 06:27:03.595586 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0806 06:27:03.595588 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:27:03.595597 (+ 9us) server_negotiation.cc:407] Connection header received
0806 06:27:03.595666 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:27:03.595666 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:27:03.595677 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:27:03.595735 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:27:03.596014 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:27:03.596506 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:27:03.597137 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:27:03.597310 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:27:03.597350 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:27:03.599622 (+ 2272us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.12.129:51549: BlockingRecv error: recv got EOF from 127.21.12.129:51549 (error 108)
Metrics: {"server-negotiator.queue_time_us":89,"thread_start_us":53,"threads_started":1}
Aug 06 06:27:03 dist-test-slave-4tdj krb5kdc[931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.12.130@KRBTEST.COM for oryx/127.21.12.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 06:27:03 dist-test-slave-4tdj krb5kdc[931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.12.130@KRBTEST.COM for oryx/127.21.12.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 06:27:03.732307 2498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:27:03.727794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:46519 (local address 127.21.12.130:38179)
0806 06:27:03.727927 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0806 06:27:03.727940 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 06:27:03.728044 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:27:03.728188 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:27:03.728189 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:27:03.728357 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0806 06:27:03.728403 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:27:03.728405 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:27:03.729050 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:27:03.729051 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:27:03.729572 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:27:03.729574 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:27:03.729888 (+ 314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:27:03.729889 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:27:03.729941 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:27:03.730406 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 06:27:03.730412 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0806 06:27:03.732235 (+ 1823us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.12.190:46519: Server oryx/127.21.12.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":69,"threads_started":1}
W20260806 06:27:03.732420 2495 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 06:27:03.727858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.130:38179 (local address 127.21.12.190:46519)
0806 06:27:03.727931 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0806 06:27:03.727934 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 06:27:03.728025 (+ 91us) server_negotiation.cc:407] Connection header received
0806 06:27:03.728115 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:27:03.728115 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:27:03.728130 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:27:03.728176 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:27:03.728468 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:27:03.728976 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:27:03.729642 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:27:03.729832 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:27:03.729886 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:27:03.732354 (+ 2468us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.12.130:38179: BlockingRecv error: recv got EOF from 127.21.12.130:38179 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Aug 06 06:27:03 dist-test-slave-4tdj krb5kdc[931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.12.131@KRBTEST.COM for oryx/127.21.12.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 06:27:03 dist-test-slave-4tdj krb5kdc[931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.12.131@KRBTEST.COM for oryx/127.21.12.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 06:27:03.865044 2499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:27:03.860738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:46519 (local address 127.21.12.131:32973)
0806 06:27:03.860864 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0806 06:27:03.860875 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0806 06:27:03.860935 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:27:03.861046 (+ 111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:27:03.861047 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:27:03.861212 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0806 06:27:03.861260 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:27:03.861261 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:27:03.861883 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:27:03.861885 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:27:03.862497 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:27:03.862500 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:27:03.862820 (+ 320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:27:03.862820 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:27:03.862858 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:27:03.863314 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 06:27:03.863319 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0806 06:27:03.864965 (+ 1646us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.12.190:46519: Server oryx/127.21.12.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":77,"threads_started":1}
W20260806 06:27:03.865111 2495 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 06:27:03.860819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.131:32973 (local address 127.21.12.190:46519)
0806 06:27:03.860878 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0806 06:27:03.860880 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:27:03.860921 (+ 41us) server_negotiation.cc:407] Connection header received
0806 06:27:03.860976 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:27:03.860977 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:27:03.860990 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:27:03.861033 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:27:03.861309 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:27:03.861794 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:27:03.862572 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:27:03.862764 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:27:03.862817 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:27:03.865051 (+ 2234us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.12.131:32973: BlockingRecv error: recv got EOF from 127.21.12.131:32973 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Aug 06 06:27:04 dist-test-slave-4tdj krb5kdc[931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.12.129@KRBTEST.COM for oryx/127.21.12.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 06:27:04 dist-test-slave-4tdj krb5kdc[931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.12.129@KRBTEST.COM for oryx/127.21.12.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 06:27:04.604128 2500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:27:04.600017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:46519 (local address 127.21.12.129:49475)
0806 06:27:04.600156 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0806 06:27:04.600167 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0806 06:27:04.600231 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:27:04.600367 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:27:04.600368 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:27:04.600524 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0806 06:27:04.600572 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:27:04.600573 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:27:04.601186 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:27:04.601187 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:27:04.601691 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:27:04.601693 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:27:04.601967 (+ 274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:27:04.601968 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:27:04.602003 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:27:04.602443 (+ 440us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 06:27:04.602448 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0806 06:27:04.604070 (+ 1622us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.12.190:46519: Server oryx/127.21.12.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":70,"threads_started":1}
W20260806 06:27:04.604230 2503 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 06:27:04.600084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.129:49475 (local address 127.21.12.190:46519)
0806 06:27:04.600183 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0806 06:27:04.600184 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0806 06:27:04.600220 (+ 36us) server_negotiation.cc:407] Connection header received
0806 06:27:04.600278 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:27:04.600279 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:27:04.600289 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:27:04.600361 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:27:04.600628 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:27:04.601124 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:27:04.601741 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:27:04.601923 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:27:04.601966 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:27:04.604151 (+ 2185us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.12.129:49475: BlockingRecv error: recv got EOF from 127.21.12.129:49475 (error 108)
Metrics: {"server-negotiator.queue_time_us":81,"thread_start_us":47,"threads_started":1}
Aug 06 06:27:04 dist-test-slave-4tdj krb5kdc[931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.12.130@KRBTEST.COM for oryx/127.21.12.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 06:27:04 dist-test-slave-4tdj krb5kdc[931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.12.130@KRBTEST.COM for oryx/127.21.12.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 06:27:04.736974 2506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:27:04.732726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:46519 (local address 127.21.12.130:49149)
0806 06:27:04.732849 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0806 06:27:04.732859 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0806 06:27:04.732914 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:27:04.733057 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:27:04.733058 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:27:04.733212 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0806 06:27:04.733260 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:27:04.733261 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:27:04.733889 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:27:04.733890 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:27:04.734463 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:27:04.734466 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:27:04.734822 (+ 356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:27:04.734823 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:27:04.734862 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:27:04.735254 (+ 392us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 06:27:04.735258 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0806 06:27:04.736913 (+ 1655us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.12.190:46519: Server oryx/127.21.12.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":57,"threads_started":1}
W20260806 06:27:04.737074 2503 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 06:27:04.732792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.130:49149 (local address 127.21.12.190:46519)
0806 06:27:04.732847 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0806 06:27:04.732849 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:27:04.732910 (+ 61us) server_negotiation.cc:407] Connection header received
0806 06:27:04.732955 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:27:04.732955 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:27:04.732989 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:27:04.733054 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:27:04.733311 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:27:04.733816 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:27:04.734560 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:27:04.734763 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:27:04.734823 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:27:04.737003 (+ 2180us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.12.130:49149: BlockingRecv error: recv got EOF from 127.21.12.130:49149 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
Aug 06 06:27:04 dist-test-slave-4tdj krb5kdc[931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.12.131@KRBTEST.COM for oryx/127.21.12.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 06:27:04 dist-test-slave-4tdj krb5kdc[931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.12.131@KRBTEST.COM for oryx/127.21.12.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 06:27:04.869643 2508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:27:04.865486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:46519 (local address 127.21.12.131:51679)
0806 06:27:04.865599 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0806 06:27:04.865610 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0806 06:27:04.865674 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:27:04.865778 (+ 104us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:27:04.865779 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:27:04.865932 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0806 06:27:04.865980 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:27:04.865981 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:27:04.866666 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:27:04.866667 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:27:04.867184 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:27:04.867186 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:27:04.867457 (+ 271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:27:04.867457 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:27:04.867493 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:27:04.867965 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 06:27:04.867969 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0806 06:27:04.869578 (+ 1609us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.12.190:46519: Server oryx/127.21.12.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":57,"threads_started":1}
W20260806 06:27:04.869742 2503 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 06:27:04.865539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.131:51679 (local address 127.21.12.190:46519)
0806 06:27:04.865610 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0806 06:27:04.865612 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:27:04.865660 (+ 48us) server_negotiation.cc:407] Connection header received
0806 06:27:04.865715 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:27:04.865715 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:27:04.865728 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:27:04.865767 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:27:04.866070 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:27:04.866584 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:27:04.867238 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:27:04.867409 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:27:04.867453 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:27:04.869682 (+ 2229us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.12.131:51679: BlockingRecv error: recv got EOF from 127.21.12.131:51679 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Aug 06 06:27:05 dist-test-slave-4tdj krb5kdc[931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.12.129@KRBTEST.COM for oryx/127.21.12.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 06:27:05 dist-test-slave-4tdj krb5kdc[931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.12.129@KRBTEST.COM for oryx/127.21.12.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 06:27:05.612912 2509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:27:05.604573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:46519 (local address 127.21.12.129:47871)
0806 06:27:05.604691 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0806 06:27:05.604706 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 06:27:05.604772 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:27:05.604910 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:27:05.604911 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:27:05.605082 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0806 06:27:05.605132 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:27:05.605134 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:27:05.605795 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:27:05.605796 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:27:05.606365 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:27:05.606367 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:27:05.606685 (+ 318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:27:05.606686 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:27:05.606741 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:27:05.611131 (+ 4390us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 06:27:05.611136 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0806 06:27:05.612855 (+ 1719us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.12.190:46519: Server oryx/127.21.12.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":57,"threads_started":1}
W20260806 06:27:05.613029 2510 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 06:27:05.604661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.129:47871 (local address 127.21.12.190:46519)
0806 06:27:05.604800 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0806 06:27:05.604802 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:27:05.604809 (+ 7us) server_negotiation.cc:407] Connection header received
0806 06:27:05.604830 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:27:05.604831 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:27:05.604842 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:27:05.604899 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:27:05.605184 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:27:05.605705 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:27:05.606425 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:27:05.606620 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:27:05.606665 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:27:05.612942 (+ 6277us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.12.129:47871: BlockingRecv error: recv got EOF from 127.21.12.129:47871 (error 108)
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":74,"threads_started":1}
Aug 06 06:27:05 dist-test-slave-4tdj krb5kdc[931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.12.130@KRBTEST.COM for oryx/127.21.12.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 06:27:05 dist-test-slave-4tdj krb5kdc[931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.12.130@KRBTEST.COM for oryx/127.21.12.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 06:27:05.741791 2511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:27:05.737425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:46519 (local address 127.21.12.130:51339)
0806 06:27:05.737550 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0806 06:27:05.737560 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0806 06:27:05.737613 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:27:05.737734 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:27:05.737735 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:27:05.737889 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0806 06:27:05.737938 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:27:05.737940 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:27:05.738691 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:27:05.738692 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:27:05.739236 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:27:05.739238 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:27:05.739582 (+ 344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:27:05.739583 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:27:05.739618 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:27:05.740021 (+ 403us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 06:27:05.740025 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0806 06:27:05.741732 (+ 1707us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.12.190:46519: Server oryx/127.21.12.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":58,"threads_started":1}
W20260806 06:27:05.741875 2510 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 06:27:05.737578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.130:51339 (local address 127.21.12.190:46519)
0806 06:27:05.737630 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0806 06:27:05.737632 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:27:05.737639 (+ 7us) server_negotiation.cc:407] Connection header received
0806 06:27:05.737665 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:27:05.737666 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:27:05.737677 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:27:05.737723 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:27:05.738059 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:27:05.738619 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:27:05.739309 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:27:05.739494 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:27:05.739575 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:27:05.741816 (+ 2241us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.12.130:51339: BlockingRecv error: recv got EOF from 127.21.12.130:51339 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
Aug 06 06:27:05 dist-test-slave-4tdj krb5kdc[931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.12.131@KRBTEST.COM for oryx/127.21.12.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 06:27:05 dist-test-slave-4tdj krb5kdc[931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.12.131@KRBTEST.COM for oryx/127.21.12.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 06:27:05.874624 2512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:27:05.870164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:46519 (local address 127.21.12.131:49113)
0806 06:27:05.870331 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0806 06:27:05.870342 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0806 06:27:05.870406 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:27:05.870534 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:27:05.870535 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:27:05.870734 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0806 06:27:05.870782 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:27:05.870784 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:27:05.871437 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:27:05.871438 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:27:05.871959 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:27:05.871960 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:27:05.872289 (+ 329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:27:05.872290 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:27:05.872342 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:27:05.872799 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 06:27:05.872804 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0806 06:27:05.874559 (+ 1755us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.12.190:46519: Server oryx/127.21.12.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":89,"threads_started":1}
W20260806 06:27:05.874758 2510 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 06:27:05.870225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.131:49113 (local address 127.21.12.190:46519)
0806 06:27:05.870270 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0806 06:27:05.870273 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 06:27:05.870400 (+ 127us) server_negotiation.cc:407] Connection header received
0806 06:27:05.870465 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:27:05.870466 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:27:05.870481 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:27:05.870528 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:27:05.870856 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:27:05.871364 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:27:05.872031 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:27:05.872229 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:27:05.872284 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:27:05.874690 (+ 2406us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.12.131:49113: BlockingRecv error: recv got EOF from 127.21.12.131:49113 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
Aug 06 06:27:06 dist-test-slave-4tdj krb5kdc[931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.12.129@KRBTEST.COM for oryx/127.21.12.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 06:27:06 dist-test-slave-4tdj krb5kdc[931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.12.129@KRBTEST.COM for oryx/127.21.12.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 06:27:06.617842 2514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:27:06.613344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:46519 (local address 127.21.12.129:45535)
0806 06:27:06.613465 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0806 06:27:06.613475 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0806 06:27:06.613547 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:27:06.613750 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:27:06.613750 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:27:06.613909 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0806 06:27:06.613960 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:27:06.613961 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:27:06.614887 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:27:06.614888 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:27:06.615408 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:27:06.615409 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:27:06.615728 (+ 319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:27:06.615729 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:27:06.615768 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:27:06.616158 (+ 390us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 06:27:06.616163 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0806 06:27:06.617780 (+ 1617us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.12.190:46519: Server oryx/127.21.12.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":65,"threads_started":1}
W20260806 06:27:06.617972 2516 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 06:27:06.613431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.129:45535 (local address 127.21.12.190:46519)
0806 06:27:06.613611 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0806 06:27:06.613613 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:27:06.613623 (+ 10us) server_negotiation.cc:407] Connection header received
0806 06:27:06.613645 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:27:06.613646 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:27:06.613658 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:27:06.613745 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:27:06.614014 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:27:06.614813 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:27:06.615474 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:27:06.615675 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:27:06.615725 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:27:06.617892 (+ 2167us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.12.129:45535: BlockingRecv error: recv got EOF from 127.21.12.129:45535 (error 108)
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":103,"threads_started":1}
Aug 06 06:27:06 dist-test-slave-4tdj krb5kdc[931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.12.130@KRBTEST.COM for oryx/127.21.12.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 06:27:06 dist-test-slave-4tdj krb5kdc[931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.12.130@KRBTEST.COM for oryx/127.21.12.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 06:27:06.746554 2517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:27:06.742276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:46519 (local address 127.21.12.130:37761)
0806 06:27:06.742420 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0806 06:27:06.742433 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 06:27:06.742501 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:27:06.742639 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:27:06.742640 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:27:06.742830 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0806 06:27:06.742891 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:27:06.742893 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:27:06.743501 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:27:06.743502 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:27:06.744042 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:27:06.744044 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:27:06.744317 (+ 273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:27:06.744317 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:27:06.744362 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:27:06.744750 (+ 388us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 06:27:06.744754 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0806 06:27:06.746475 (+ 1721us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.12.190:46519: Server oryx/127.21.12.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":78,"threads_started":1}
W20260806 06:27:06.746649 2516 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 06:27:06.742348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.130:37761 (local address 127.21.12.190:46519)
0806 06:27:06.742395 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0806 06:27:06.742396 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0806 06:27:06.742495 (+ 99us) server_negotiation.cc:407] Connection header received
0806 06:27:06.742565 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:27:06.742566 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:27:06.742579 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:27:06.742626 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:27:06.742955 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:27:06.743444 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:27:06.744099 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:27:06.744271 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:27:06.744315 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:27:06.746593 (+ 2278us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.12.130:37761: BlockingRecv error: recv got EOF from 127.21.12.130:37761 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
Aug 06 06:27:06 dist-test-slave-4tdj krb5kdc[931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.12.131@KRBTEST.COM for oryx/127.21.12.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 06:27:06 dist-test-slave-4tdj krb5kdc[931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.12.131@KRBTEST.COM for oryx/127.21.12.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 06:27:06.879463 2518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:27:06.875128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:46519 (local address 127.21.12.131:35989)
0806 06:27:06.875268 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0806 06:27:06.875280 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 06:27:06.875339 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:27:06.875460 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:27:06.875461 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:27:06.875628 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0806 06:27:06.875674 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:27:06.875676 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:27:06.876267 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:27:06.876268 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:27:06.876823 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:27:06.876826 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:27:06.877150 (+ 324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:27:06.877151 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:27:06.877204 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:27:06.877617 (+ 413us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 06:27:06.877622 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0806 06:27:06.879404 (+ 1782us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.12.190:46519: Server oryx/127.21.12.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":78,"threads_started":1}
W20260806 06:27:06.879547 2516 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 06:27:06.875197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.131:35989 (local address 127.21.12.190:46519)
0806 06:27:06.875283 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0806 06:27:06.875285 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:27:06.875372 (+ 87us) server_negotiation.cc:407] Connection header received
0806 06:27:06.875394 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:27:06.875395 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:27:06.875408 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:27:06.875448 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:27:06.875725 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:27:06.876210 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:27:06.876884 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:27:06.877068 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:27:06.877116 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:27:06.879495 (+ 2379us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.12.131:35989: BlockingRecv error: recv got EOF from 127.21.12.131:35989 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Aug 06 06:27:07 dist-test-slave-4tdj krb5kdc[931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.12.129@KRBTEST.COM for oryx/127.21.12.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 06:27:07 dist-test-slave-4tdj krb5kdc[931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.12.129@KRBTEST.COM for oryx/127.21.12.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 06:27:07.622836 2519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:27:07.618391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:46519 (local address 127.21.12.129:49285)
0806 06:27:07.618514 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0806 06:27:07.618582 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0806 06:27:07.618637 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:27:07.618796 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:27:07.618797 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:27:07.619021 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0806 06:27:07.619076 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:27:07.619078 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:27:07.619702 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:27:07.619703 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:27:07.620226 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:27:07.620227 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:27:07.620543 (+ 316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:27:07.620544 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:27:07.620585 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:27:07.621018 (+ 433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 06:27:07.621022 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0806 06:27:07.622777 (+ 1755us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.12.190:46519: Server oryx/127.21.12.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":66,"threads_started":1}
W20260806 06:27:07.622933 2520 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 06:27:07.618452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.129:49285 (local address 127.21.12.190:46519)
0806 06:27:07.618622 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0806 06:27:07.618624 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:27:07.618633 (+ 9us) server_negotiation.cc:407] Connection header received
0806 06:27:07.618697 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:27:07.618697 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:27:07.618711 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:27:07.618783 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:27:07.619143 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:27:07.619643 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:27:07.620292 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:27:07.620487 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:27:07.620586 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:27:07.622866 (+ 2280us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.12.129:49285: BlockingRecv error: recv got EOF from 127.21.12.129:49285 (error 108)
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":110,"threads_started":1}
Aug 06 06:27:07 dist-test-slave-4tdj krb5kdc[931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.12.130@KRBTEST.COM for oryx/127.21.12.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 06:27:07 dist-test-slave-4tdj krb5kdc[931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.12.130@KRBTEST.COM for oryx/127.21.12.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 06:27:07.751259 2521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:27:07.747011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:46519 (local address 127.21.12.130:49745)
0806 06:27:07.747163 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0806 06:27:07.747174 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0806 06:27:07.747233 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:27:07.747343 (+ 110us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:27:07.747344 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:27:07.747496 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0806 06:27:07.747548 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:27:07.747549 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:27:07.748193 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:27:07.748193 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:27:07.748713 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:27:07.748714 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:27:07.749035 (+ 321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:27:07.749035 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:27:07.749077 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:27:07.749493 (+ 416us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 06:27:07.749496 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0806 06:27:07.751200 (+ 1704us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.12.190:46519: Server oryx/127.21.12.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":86,"threads_started":1}
W20260806 06:27:07.751406 2520 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 06:27:07.747082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.130:49745 (local address 127.21.12.190:46519)
0806 06:27:07.747149 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0806 06:27:07.747152 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 06:27:07.747224 (+ 72us) server_negotiation.cc:407] Connection header received
0806 06:27:07.747281 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:27:07.747282 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:27:07.747293 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:27:07.747337 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:27:07.747621 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:27:07.748125 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:27:07.748780 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:27:07.748973 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:27:07.749028 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:27:07.751334 (+ 2306us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.12.130:49745: BlockingRecv error: recv got EOF from 127.21.12.130:49745 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Aug 06 06:27:07 dist-test-slave-4tdj krb5kdc[931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.12.131@KRBTEST.COM for oryx/127.21.12.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 06:27:07 dist-test-slave-4tdj krb5kdc[931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.12.131@KRBTEST.COM for oryx/127.21.12.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 06:27:07.884125 2523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:27:07.879926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:46519 (local address 127.21.12.131:42115)
0806 06:27:07.880077 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0806 06:27:07.880088 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0806 06:27:07.880145 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:27:07.880260 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:27:07.880261 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:27:07.880421 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0806 06:27:07.880471 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:27:07.880472 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:27:07.881071 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:27:07.881072 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:27:07.881595 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:27:07.881597 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:27:07.881871 (+ 274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:27:07.881872 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:27:07.881908 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:27:07.882377 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 06:27:07.882382 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0806 06:27:07.884049 (+ 1667us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.12.190:46519: Server oryx/127.21.12.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":77,"threads_started":1}
W20260806 06:27:07.884203 2520 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 06:27:07.880002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.131:42115 (local address 127.21.12.190:46519)
0806 06:27:07.880056 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0806 06:27:07.880058 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:27:07.880140 (+ 82us) server_negotiation.cc:407] Connection header received
0806 06:27:07.880185 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:27:07.880185 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:27:07.880198 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:27:07.880249 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:27:07.880518 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:27:07.881002 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:27:07.881655 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:27:07.881826 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:27:07.881867 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:27:07.884141 (+ 2274us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.12.131:42115: BlockingRecv error: recv got EOF from 127.21.12.131:42115 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
Aug 06 06:27:08 dist-test-slave-4tdj krb5kdc[931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.12.129@KRBTEST.COM for oryx/127.21.12.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 06:27:08 dist-test-slave-4tdj krb5kdc[931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.12.129@KRBTEST.COM for oryx/127.21.12.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 06:27:08.627715 2526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:27:08.623264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:46519 (local address 127.21.12.129:57919)
0806 06:27:08.623413 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0806 06:27:08.623424 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0806 06:27:08.623474 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:27:08.623630 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:27:08.623631 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:27:08.623797 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0806 06:27:08.623848 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:27:08.623849 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:27:08.624512 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:27:08.624514 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:27:08.625035 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:27:08.625037 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:27:08.625360 (+ 323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:27:08.625361 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:27:08.625399 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:27:08.625909 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 06:27:08.625914 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0806 06:27:08.627657 (+ 1743us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.12.190:46519: Server oryx/127.21.12.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":60,"threads_started":1}
W20260806 06:27:08.627823 2527 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 06:27:08.623416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.129:57919 (local address 127.21.12.190:46519)
0806 06:27:08.623516 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0806 06:27:08.623518 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:27:08.623526 (+ 8us) server_negotiation.cc:407] Connection header received
0806 06:27:08.623550 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:27:08.623551 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:27:08.623562 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:27:08.623619 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:27:08.623933 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:27:08.624439 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:27:08.625108 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:27:08.625307 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:27:08.625357 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:27:08.627755 (+ 2398us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.12.129:57919: BlockingRecv error: recv got EOF from 127.21.12.129:57919 (error 108)
Metrics: {"server-negotiator.queue_time_us":82,"thread_start_us":48,"threads_started":1}
Aug 06 06:27:08 dist-test-slave-4tdj krb5kdc[931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.12.130@KRBTEST.COM for oryx/127.21.12.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 06:27:08 dist-test-slave-4tdj krb5kdc[931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.12.130@KRBTEST.COM for oryx/127.21.12.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 06:27:08.756038 2529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:27:08.751674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:46519 (local address 127.21.12.130:57937)
0806 06:27:08.751809 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0806 06:27:08.751822 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 06:27:08.751900 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:27:08.752033 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:27:08.752035 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:27:08.752203 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0806 06:27:08.752250 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:27:08.752252 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:27:08.753034 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:27:08.753036 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:27:08.753555 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:27:08.753557 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:27:08.753878 (+ 321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:27:08.753878 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:27:08.753929 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:27:08.754400 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 06:27:08.754405 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0806 06:27:08.755972 (+ 1567us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.12.190:46519: Server oryx/127.21.12.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":63,"threads_started":1}
W20260806 06:27:08.756127 2527 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 06:27:08.751753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.130:57937 (local address 127.21.12.190:46519)
0806 06:27:08.751835 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0806 06:27:08.751837 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:27:08.751888 (+ 51us) server_negotiation.cc:407] Connection header received
0806 06:27:08.751959 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:27:08.751960 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:27:08.751973 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:27:08.752024 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:27:08.752321 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:27:08.752961 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:27:08.753636 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:27:08.753819 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:27:08.753872 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:27:08.756068 (+ 2196us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.12.130:57937: BlockingRecv error: recv got EOF from 127.21.12.130:57937 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Aug 06 06:27:08 dist-test-slave-4tdj krb5kdc[931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.12.131@KRBTEST.COM for oryx/127.21.12.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 06:27:08 dist-test-slave-4tdj krb5kdc[931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.12.131@KRBTEST.COM for oryx/127.21.12.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 06:27:08.889165 2530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:27:08.884596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:46519 (local address 127.21.12.131:44571)
0806 06:27:08.884775 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0806 06:27:08.884786 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0806 06:27:08.884858 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:27:08.884985 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:27:08.884986 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:27:08.885157 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0806 06:27:08.885222 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:27:08.885223 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:27:08.885895 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:27:08.885896 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:27:08.886473 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:27:08.886476 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:27:08.886857 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:27:08.886858 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:27:08.886924 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:27:08.887363 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 06:27:08.887368 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0806 06:27:08.889105 (+ 1737us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.12.190:46519: Server oryx/127.21.12.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":85,"threads_started":1}
W20260806 06:27:08.889252 2527 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 06:27:08.884724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.131:44571 (local address 127.21.12.190:46519)
0806 06:27:08.884773 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0806 06:27:08.884775 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:27:08.884838 (+ 63us) server_negotiation.cc:407] Connection header received
0806 06:27:08.884905 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:27:08.884906 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:27:08.884918 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:27:08.884963 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:27:08.885310 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:27:08.885825 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:27:08.886553 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:27:08.886788 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:27:08.886843 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:27:08.889189 (+ 2346us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.12.131:44571: BlockingRecv error: recv got EOF from 127.21.12.131:44571 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
Aug 06 06:27:09 dist-test-slave-4tdj krb5kdc[931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.12.129@KRBTEST.COM for oryx/127.21.12.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 06:27:09 dist-test-slave-4tdj krb5kdc[931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.12.129@KRBTEST.COM for oryx/127.21.12.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 06:27:09.632493 2533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:27:09.628142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:46519 (local address 127.21.12.129:43245)
0806 06:27:09.628267 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0806 06:27:09.628279 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 06:27:09.628331 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:27:09.628495 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:27:09.628496 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:27:09.628670 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0806 06:27:09.628720 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:27:09.628722 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:27:09.629359 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:27:09.629359 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:27:09.629875 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:27:09.629876 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:27:09.630203 (+ 327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:27:09.630204 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:27:09.630241 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:27:09.630706 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 06:27:09.630712 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0806 06:27:09.632423 (+ 1711us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.12.190:46519: Server oryx/127.21.12.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":62,"threads_started":1}
W20260806 06:27:09.632586 2534 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 06:27:09.628197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.129:43245 (local address 127.21.12.190:46519)
0806 06:27:09.628306 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0806 06:27:09.628308 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:27:09.628320 (+ 12us) server_negotiation.cc:407] Connection header received
0806 06:27:09.628389 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:27:09.628390 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:27:09.628400 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:27:09.628512 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:27:09.628791 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:27:09.629300 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:27:09.629945 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:27:09.630148 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:27:09.630201 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:27:09.632514 (+ 2313us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.12.129:43245: BlockingRecv error: recv got EOF from 127.21.12.129:43245 (error 108)
Metrics: {"server-negotiator.queue_time_us":89,"thread_start_us":57,"threads_started":1}
Aug 06 06:27:09 dist-test-slave-4tdj krb5kdc[931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.12.130@KRBTEST.COM for oryx/127.21.12.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 06:27:09 dist-test-slave-4tdj krb5kdc[931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.12.130@KRBTEST.COM for oryx/127.21.12.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 06:27:09.760882 2535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:27:09.756473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:46519 (local address 127.21.12.130:56971)
0806 06:27:09.756642 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0806 06:27:09.756655 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 06:27:09.756728 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:27:09.756859 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:27:09.756860 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:27:09.757039 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0806 06:27:09.757085 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:27:09.757087 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:27:09.757716 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:27:09.757717 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:27:09.758237 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:27:09.758238 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:27:09.758598 (+ 360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:27:09.758599 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:27:09.758657 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:27:09.759093 (+ 436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 06:27:09.759098 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0806 06:27:09.760804 (+ 1706us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.12.190:46519: Server oryx/127.21.12.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":91,"threads_started":1}
W20260806 06:27:09.760959 2534 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 06:27:09.756592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.130:56971 (local address 127.21.12.190:46519)
0806 06:27:09.756649 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0806 06:27:09.756651 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:27:09.756708 (+ 57us) server_negotiation.cc:407] Connection header received
0806 06:27:09.756792 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:27:09.756793 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:27:09.756807 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:27:09.756848 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:27:09.757157 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:27:09.757644 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:27:09.758342 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:27:09.758538 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:27:09.758594 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:27:09.760905 (+ 2311us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.12.130:56971: BlockingRecv error: recv got EOF from 127.21.12.130:56971 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
Aug 06 06:27:09 dist-test-slave-4tdj krb5kdc[931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.12.131@KRBTEST.COM for oryx/127.21.12.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 06:27:09 dist-test-slave-4tdj krb5kdc[931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.12.131@KRBTEST.COM for oryx/127.21.12.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 06:27:09.894168 2536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:27:09.889639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:46519 (local address 127.21.12.131:38449)
0806 06:27:09.889763 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0806 06:27:09.889778 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 06:27:09.889849 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:27:09.889980 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:27:09.889981 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:27:09.890170 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0806 06:27:09.890215 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:27:09.890217 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:27:09.890870 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:27:09.890871 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:27:09.891420 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:27:09.891423 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:27:09.891749 (+ 326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:27:09.891751 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:27:09.891809 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:27:09.892244 (+ 435us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 06:27:09.892250 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0806 06:27:09.894108 (+ 1858us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.12.190:46519: Server oryx/127.21.12.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":65,"threads_started":1}
W20260806 06:27:09.894284 2534 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 06:27:09.889728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.131:38449 (local address 127.21.12.190:46519)
0806 06:27:09.889792 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0806 06:27:09.889794 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:27:09.889830 (+ 36us) server_negotiation.cc:407] Connection header received
0806 06:27:09.889905 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:27:09.889906 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:27:09.889920 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:27:09.889968 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:27:09.890286 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:27:09.890799 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:27:09.891486 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:27:09.891688 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:27:09.891743 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:27:09.894219 (+ 2476us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.12.131:38449: BlockingRecv error: recv got EOF from 127.21.12.131:38449 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Aug 06 06:27:10 dist-test-slave-4tdj krb5kdc[931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.12.129@KRBTEST.COM for oryx/127.21.12.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 06:27:10 dist-test-slave-4tdj krb5kdc[931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.12.129@KRBTEST.COM for oryx/127.21.12.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 06:27:10.641305 2537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:27:10.632922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:46519 (local address 127.21.12.129:37905)
0806 06:27:10.633076 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0806 06:27:10.633087 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0806 06:27:10.633135 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:27:10.633261 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:27:10.633262 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:27:10.633427 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0806 06:27:10.633476 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:27:10.633478 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:27:10.634101 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:27:10.634101 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:27:10.634697 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:27:10.634700 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:27:10.635062 (+ 362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:27:10.635063 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:27:10.635124 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:27:10.639567 (+ 4443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 06:27:10.639572 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0806 06:27:10.641243 (+ 1671us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.12.190:46519: Server oryx/127.21.12.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":59,"threads_started":1}
W20260806 06:27:10.641446 2538 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 06:27:10.632986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.129:37905 (local address 127.21.12.190:46519)
0806 06:27:10.633110 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0806 06:27:10.633112 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:27:10.633123 (+ 11us) server_negotiation.cc:407] Connection header received
0806 06:27:10.633180 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:27:10.633180 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:27:10.633191 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:27:10.633251 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:27:10.633548 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:27:10.634048 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:27:10.634785 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:27:10.634988 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:27:10.635048 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:27:10.641365 (+ 6317us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.12.129:37905: BlockingRecv error: recv got EOF from 127.21.12.129:37905 (error 108)
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":62,"threads_started":1}
Aug 06 06:27:10 dist-test-slave-4tdj krb5kdc[931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.12.130@KRBTEST.COM for oryx/127.21.12.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 06:27:10 dist-test-slave-4tdj krb5kdc[931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.12.130@KRBTEST.COM for oryx/127.21.12.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 06:27:10.766194 2539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:27:10.761374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:46519 (local address 127.21.12.130:54883)
0806 06:27:10.761541 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0806 06:27:10.761554 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 06:27:10.761626 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:27:10.761757 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:27:10.761758 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:27:10.761933 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0806 06:27:10.761981 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:27:10.761983 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:27:10.762677 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:27:10.762678 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:27:10.763205 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:27:10.763206 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:27:10.763522 (+ 316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:27:10.763523 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:27:10.763578 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:27:10.764030 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 06:27:10.764035 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0806 06:27:10.766126 (+ 2091us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.12.190:46519: Server oryx/127.21.12.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":86,"threads_started":1}
W20260806 06:27:10.766275 2538 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 06:27:10.761490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.130:54883 (local address 127.21.12.190:46519)
0806 06:27:10.761548 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0806 06:27:10.761550 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:27:10.761606 (+ 56us) server_negotiation.cc:407] Connection header received
0806 06:27:10.761681 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:27:10.761682 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:27:10.761696 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:27:10.761744 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:27:10.762052 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:27:10.762600 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:27:10.763279 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:27:10.763463 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:27:10.763518 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:27:10.766220 (+ 2702us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.12.130:54883: BlockingRecv error: recv got EOF from 127.21.12.130:54883 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
Aug 06 06:27:10 dist-test-slave-4tdj krb5kdc[931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.12.131@KRBTEST.COM for oryx/127.21.12.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 06:27:10 dist-test-slave-4tdj krb5kdc[931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.12.131@KRBTEST.COM for oryx/127.21.12.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 06:27:10.898845 2543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:27:10.894717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:46519 (local address 127.21.12.131:36917)
0806 06:27:10.894871 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0806 06:27:10.894884 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 06:27:10.894943 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:27:10.895074 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:27:10.895076 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:27:10.895239 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0806 06:27:10.895287 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:27:10.895289 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:27:10.895892 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:27:10.895893 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:27:10.896416 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:27:10.896417 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:27:10.896729 (+ 312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:27:10.896729 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:27:10.896766 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:27:10.897205 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 06:27:10.897210 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0806 06:27:10.898785 (+ 1575us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.12.190:46519: Server oryx/127.21.12.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":81,"threads_started":1}
W20260806 06:27:10.898933 2538 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 06:27:10.894877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.131:36917 (local address 127.21.12.190:46519)
0806 06:27:10.894922 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0806 06:27:10.894924 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:27:10.894932 (+ 8us) server_negotiation.cc:407] Connection header received
0806 06:27:10.894996 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:27:10.894997 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:27:10.895008 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:27:10.895057 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:27:10.895338 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:27:10.895827 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:27:10.896482 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:27:10.896671 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:27:10.896721 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:27:10.898871 (+ 2150us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.12.131:36917: BlockingRecv error: recv got EOF from 127.21.12.131:36917 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
Aug 06 06:27:11 dist-test-slave-4tdj krb5kdc[931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.12.129@KRBTEST.COM for oryx/127.21.12.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 06:27:11 dist-test-slave-4tdj krb5kdc[931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.12.129@KRBTEST.COM for oryx/127.21.12.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 06:27:11.646063 2544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:27:11.641732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:46519 (local address 127.21.12.129:39265)
0806 06:27:11.641867 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0806 06:27:11.641879 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 06:27:11.641946 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:27:11.642071 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:27:11.642072 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:27:11.642254 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0806 06:27:11.642332 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:27:11.642336 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:27:11.643000 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:27:11.643001 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:27:11.643515 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:27:11.643516 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:27:11.643837 (+ 321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:27:11.643838 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:27:11.643872 (+ 34us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:27:11.644291 (+ 419us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 06:27:11.644296 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0806 06:27:11.645991 (+ 1695us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.12.190:46519: Server oryx/127.21.12.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":66,"threads_started":1}
W20260806 06:27:11.646145 2545 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 06:27:11.641781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.129:39265 (local address 127.21.12.190:46519)
0806 06:27:11.641881 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0806 06:27:11.641883 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:27:11.641931 (+ 48us) server_negotiation.cc:407] Connection header received
0806 06:27:11.641993 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:27:11.641994 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:27:11.642006 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:27:11.642060 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:27:11.642407 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:27:11.642939 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:27:11.643590 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:27:11.643785 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:27:11.643837 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:27:11.646074 (+ 2237us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.12.129:39265: BlockingRecv error: recv got EOF from 127.21.12.129:39265 (error 108)
Metrics: {"server-negotiator.queue_time_us":74,"thread_start_us":48,"threads_started":1}
Aug 06 06:27:11 dist-test-slave-4tdj krb5kdc[931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.12.130@KRBTEST.COM for oryx/127.21.12.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 06:27:11 dist-test-slave-4tdj krb5kdc[931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.12.130@KRBTEST.COM for oryx/127.21.12.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 06:27:11.771023 2547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:27:11.766658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:46519 (local address 127.21.12.130:53035)
0806 06:27:11.766804 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0806 06:27:11.766815 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0806 06:27:11.766880 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:27:11.766996 (+ 116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:27:11.766997 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:27:11.767170 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0806 06:27:11.767217 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:27:11.767219 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:27:11.767817 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:27:11.767817 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:27:11.768327 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:27:11.768329 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:27:11.768601 (+ 272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:27:11.768601 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:27:11.768637 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:27:11.769096 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 06:27:11.769101 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0806 06:27:11.770950 (+ 1849us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.12.190:46519: Server oryx/127.21.12.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":74,"threads_started":1}
W20260806 06:27:11.771103 2545 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 06:27:11.766748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.130:53035 (local address 127.21.12.190:46519)
0806 06:27:11.766811 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0806 06:27:11.766813 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:27:11.766873 (+ 60us) server_negotiation.cc:407] Connection header received
0806 06:27:11.766935 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:27:11.766936 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:27:11.766946 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:27:11.766983 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:27:11.767268 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:27:11.767758 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:27:11.768380 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:27:11.768556 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:27:11.768600 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:27:11.771047 (+ 2447us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.12.130:53035: BlockingRecv error: recv got EOF from 127.21.12.130:53035 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Aug 06 06:27:11 dist-test-slave-4tdj krb5kdc[931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.12.131@KRBTEST.COM for oryx/127.21.12.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 06:27:11 dist-test-slave-4tdj krb5kdc[931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.12.131@KRBTEST.COM for oryx/127.21.12.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 06:27:11.903399 2548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:27:11.899270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:46519 (local address 127.21.12.131:35673)
0806 06:27:11.899392 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0806 06:27:11.899402 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0806 06:27:11.899458 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:27:11.899596 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:27:11.899597 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:27:11.899777 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0806 06:27:11.899825 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:27:11.899827 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:27:11.900409 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:27:11.900410 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:27:11.900940 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:27:11.900941 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:27:11.901218 (+ 277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:27:11.901219 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:27:11.901256 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:27:11.901726 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 06:27:11.901732 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0806 06:27:11.903332 (+ 1600us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.12.190:46519: Server oryx/127.21.12.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":59,"threads_started":1}
W20260806 06:27:11.903501 2545 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 06:27:11.899312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.131:35673 (local address 127.21.12.190:46519)
0806 06:27:11.899391 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0806 06:27:11.899393 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:27:11.899452 (+ 59us) server_negotiation.cc:407] Connection header received
0806 06:27:11.899502 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:27:11.899503 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:27:11.899515 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:27:11.899585 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:27:11.899875 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:27:11.900358 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:27:11.900994 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:27:11.901172 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:27:11.901215 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:27:11.903436 (+ 2221us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.12.131:35673: BlockingRecv error: recv got EOF from 127.21.12.131:35673 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Aug 06 06:27:12 dist-test-slave-4tdj krb5kdc[931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.12.129@KRBTEST.COM for oryx/127.21.12.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 06:27:12 dist-test-slave-4tdj krb5kdc[931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.12.129@KRBTEST.COM for oryx/127.21.12.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 06:27:12.650795 2549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:27:12.646520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:46519 (local address 127.21.12.129:56281)
0806 06:27:12.646649 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0806 06:27:12.646660 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0806 06:27:12.646708 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:27:12.646847 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:27:12.646847 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:27:12.647030 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0806 06:27:12.647079 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:27:12.647081 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:27:12.647711 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:27:12.647712 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:27:12.648246 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:27:12.648248 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:27:12.648585 (+ 337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:27:12.648586 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:27:12.648621 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:27:12.649034 (+ 413us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 06:27:12.649038 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0806 06:27:12.650727 (+ 1689us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.12.190:46519: Server oryx/127.21.12.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":60,"threads_started":1}
W20260806 06:27:12.650919 2550 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 06:27:12.646591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.129:56281 (local address 127.21.12.190:46519)
0806 06:27:12.646702 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0806 06:27:12.646704 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:27:12.646712 (+ 8us) server_negotiation.cc:407] Connection header received
0806 06:27:12.646769 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:27:12.646770 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:27:12.646787 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:27:12.646836 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:27:12.647138 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:27:12.647655 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:27:12.648314 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:27:12.648531 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:27:12.648584 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:27:12.650847 (+ 2263us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.12.129:56281: BlockingRecv error: recv got EOF from 127.21.12.129:56281 (error 108)
Metrics: {"server-negotiator.queue_time_us":91,"thread_start_us":53,"threads_started":1}
Aug 06 06:27:12 dist-test-slave-4tdj krb5kdc[931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.12.130@KRBTEST.COM for oryx/127.21.12.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 06:27:12 dist-test-slave-4tdj krb5kdc[931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.12.130@KRBTEST.COM for oryx/127.21.12.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 06:27:12.776165 2551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:27:12.771494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:46519 (local address 127.21.12.130:42247)
0806 06:27:12.771633 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0806 06:27:12.771644 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0806 06:27:12.771716 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:27:12.771867 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:27:12.771868 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:27:12.772062 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0806 06:27:12.772109 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:27:12.772112 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:27:12.772910 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:27:12.772911 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:27:12.773487 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:27:12.773489 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:27:12.773820 (+ 331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:27:12.773821 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:27:12.773876 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:27:12.774356 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 06:27:12.774361 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0806 06:27:12.776107 (+ 1746us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.12.190:46519: Server oryx/127.21.12.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":79,"threads_started":1}
W20260806 06:27:12.776271 2550 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 06:27:12.771570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.130:42247 (local address 127.21.12.190:46519)
0806 06:27:12.771630 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0806 06:27:12.771633 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 06:27:12.771699 (+ 66us) server_negotiation.cc:407] Connection header received
0806 06:27:12.771777 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:27:12.771778 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:27:12.771792 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:27:12.771844 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:27:12.772177 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:27:12.772837 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:27:12.773560 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:27:12.773762 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:27:12.773816 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:27:12.776201 (+ 2385us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.12.130:42247: BlockingRecv error: recv got EOF from 127.21.12.130:42247 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
Aug 06 06:27:12 dist-test-slave-4tdj krb5kdc[931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.12.131@KRBTEST.COM for oryx/127.21.12.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 06:27:12 dist-test-slave-4tdj krb5kdc[931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.12.131@KRBTEST.COM for oryx/127.21.12.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 06:27:12.908041 2554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:27:12.903860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:46519 (local address 127.21.12.131:60143)
0806 06:27:12.904002 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0806 06:27:12.904013 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0806 06:27:12.904075 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:27:12.904195 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:27:12.904196 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:27:12.904364 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0806 06:27:12.904410 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:27:12.904411 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:27:12.905014 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:27:12.905014 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:27:12.905542 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:27:12.905543 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:27:12.905812 (+ 269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:27:12.905812 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:27:12.905848 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:27:12.906253 (+ 405us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 06:27:12.906257 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0806 06:27:12.907972 (+ 1715us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.12.190:46519: Server oryx/127.21.12.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":67,"threads_started":1}
W20260806 06:27:12.908126 2550 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 06:27:12.903967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.131:60143 (local address 127.21.12.190:46519)
0806 06:27:12.904023 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0806 06:27:12.904025 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:27:12.904060 (+ 35us) server_negotiation.cc:407] Connection header received
0806 06:27:12.904125 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:27:12.904126 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:27:12.904138 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:27:12.904184 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:27:12.904456 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:27:12.904951 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:27:12.905598 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:27:12.905768 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:27:12.905811 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:27:12.908066 (+ 2255us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.12.131:60143: BlockingRecv error: recv got EOF from 127.21.12.131:60143 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
Aug 06 06:27:13 dist-test-slave-4tdj krb5kdc[931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.12.129@KRBTEST.COM for oryx/127.21.12.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 06:27:13 dist-test-slave-4tdj krb5kdc[931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.12.129@KRBTEST.COM for oryx/127.21.12.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 06:27:13.655825 2555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:27:13.651262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:46519 (local address 127.21.12.129:42423)
0806 06:27:13.651389 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0806 06:27:13.651402 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 06:27:13.651467 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:27:13.651628 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:27:13.651628 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:27:13.651800 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0806 06:27:13.651850 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:27:13.651852 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:27:13.652558 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:27:13.652559 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:27:13.653103 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:27:13.653105 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:27:13.653460 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:27:13.653461 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:27:13.653515 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:27:13.653947 (+ 432us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 06:27:13.653952 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0806 06:27:13.655748 (+ 1796us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.12.190:46519: Server oryx/127.21.12.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":59,"threads_started":1}
W20260806 06:27:13.655921 2556 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 06:27:13.651332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.129:42423 (local address 127.21.12.190:46519)
0806 06:27:13.651465 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0806 06:27:13.651467 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:27:13.651476 (+ 9us) server_negotiation.cc:407] Connection header received
0806 06:27:13.651545 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:27:13.651545 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:27:13.651557 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:27:13.651618 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:27:13.651943 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:27:13.652455 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:27:13.653193 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:27:13.653394 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:27:13.653449 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:27:13.655859 (+ 2410us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.12.129:42423: BlockingRecv error: recv got EOF from 127.21.12.129:42423 (error 108)
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":65,"threads_started":1}
Aug 06 06:27:13 dist-test-slave-4tdj krb5kdc[931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.12.130@KRBTEST.COM for oryx/127.21.12.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 06:27:13 dist-test-slave-4tdj krb5kdc[931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.12.130@KRBTEST.COM for oryx/127.21.12.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 06:27:13.781164 2557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:27:13.776650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:46519 (local address 127.21.12.130:32939)
0806 06:27:13.776797 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0806 06:27:13.776808 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0806 06:27:13.776858 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:27:13.777020 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:27:13.777022 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:27:13.777227 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0806 06:27:13.777285 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:27:13.777287 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:27:13.777960 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:27:13.777962 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:27:13.778526 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:27:13.778528 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:27:13.778849 (+ 321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:27:13.778850 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:27:13.778905 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:27:13.779346 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 06:27:13.779351 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0806 06:27:13.781066 (+ 1715us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.12.190:46519: Server oryx/127.21.12.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":87,"threads_started":1}
W20260806 06:27:13.781219 2556 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 06:27:13.776790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.130:32939 (local address 127.21.12.190:46519)
0806 06:27:13.776855 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0806 06:27:13.776856 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0806 06:27:13.776867 (+ 11us) server_negotiation.cc:407] Connection header received
0806 06:27:13.776896 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:27:13.776896 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:27:13.776908 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:27:13.776970 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:27:13.777370 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:27:13.777885 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:27:13.778602 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:27:13.778789 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:27:13.778844 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:27:13.781164 (+ 2320us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.12.130:32939: BlockingRecv error: recv got EOF from 127.21.12.130:32939 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Aug 06 06:27:13 dist-test-slave-4tdj krb5kdc[931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.12.131@KRBTEST.COM for oryx/127.21.12.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 06:27:13 dist-test-slave-4tdj krb5kdc[931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.12.131@KRBTEST.COM for oryx/127.21.12.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 06:27:13.913249 2558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:27:13.908517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:46519 (local address 127.21.12.131:40057)
0806 06:27:13.908661 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0806 06:27:13.908672 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0806 06:27:13.908739 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:27:13.908861 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:27:13.908862 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:27:13.909035 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0806 06:27:13.909081 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:27:13.909083 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:27:13.909724 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:27:13.909725 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:27:13.910381 (+ 656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:27:13.910383 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:27:13.910782 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:27:13.910783 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:27:13.910833 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:27:13.911341 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 06:27:13.911346 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0806 06:27:13.913173 (+ 1827us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.12.190:46519: Server oryx/127.21.12.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":84,"threads_started":1}
W20260806 06:27:13.913329 2556 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 06:27:13.908587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.131:40057 (local address 127.21.12.190:46519)
0806 06:27:13.908662 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0806 06:27:13.908664 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:27:13.908723 (+ 59us) server_negotiation.cc:407] Connection header received
0806 06:27:13.908792 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:27:13.908793 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:27:13.908807 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:27:13.908851 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:27:13.909138 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:27:13.909642 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:27:13.910481 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:27:13.910688 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:27:13.910755 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:27:13.913269 (+ 2514us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.12.131:40057: BlockingRecv error: recv got EOF from 127.21.12.131:40057 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Aug 06 06:27:14 dist-test-slave-4tdj krb5kdc[931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.12.129@KRBTEST.COM for oryx/127.21.12.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 06:27:14 dist-test-slave-4tdj krb5kdc[931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.12.129@KRBTEST.COM for oryx/127.21.12.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 06:27:14.660449 2563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:27:14.656305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:46519 (local address 127.21.12.129:45347)
0806 06:27:14.656436 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0806 06:27:14.656447 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0806 06:27:14.656498 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:27:14.656626 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:27:14.656627 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:27:14.656793 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0806 06:27:14.656844 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:27:14.656845 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:27:14.657443 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:27:14.657444 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:27:14.657957 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:27:14.657958 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:27:14.658251 (+ 293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:27:14.658251 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:27:14.658306 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:27:14.658689 (+ 383us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 06:27:14.658693 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0806 06:27:14.660371 (+ 1678us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.12.190:46519: Server oryx/127.21.12.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":60,"threads_started":1}
W20260806 06:27:14.660530 2565 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 06:27:14.656365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.129:45347 (local address 127.21.12.190:46519)
0806 06:27:14.656501 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0806 06:27:14.656502 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0806 06:27:14.656536 (+ 34us) server_negotiation.cc:407] Connection header received
0806 06:27:14.656555 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:27:14.656556 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:27:14.656566 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:27:14.656614 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:27:14.656896 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:27:14.657391 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:27:14.658025 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:27:14.658203 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:27:14.658248 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:27:14.660467 (+ 2219us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.12.129:45347: BlockingRecv error: recv got EOF from 127.21.12.129:45347 (error 108)
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":76,"threads_started":1}
Aug 06 06:27:14 dist-test-slave-4tdj krb5kdc[931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.12.130@KRBTEST.COM for oryx/127.21.12.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 06:27:14 dist-test-slave-4tdj krb5kdc[931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.12.130@KRBTEST.COM for oryx/127.21.12.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 06:27:14.786044 2567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:27:14.781694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:46519 (local address 127.21.12.130:43249)
0806 06:27:14.781841 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0806 06:27:14.781853 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 06:27:14.781917 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:27:14.782073 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:27:14.782074 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:27:14.782273 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0806 06:27:14.782357 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:27:14.782359 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:27:14.782965 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:27:14.782966 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:27:14.783504 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:27:14.783506 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:27:14.783793 (+ 287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:27:14.783794 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:27:14.783831 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:27:14.784225 (+ 394us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 06:27:14.784229 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0806 06:27:14.785951 (+ 1722us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.12.190:46519: Server oryx/127.21.12.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":86,"threads_started":1}
W20260806 06:27:14.786119 2565 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 06:27:14.781746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.130:43249 (local address 127.21.12.190:46519)
0806 06:27:14.781801 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0806 06:27:14.781804 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 06:27:14.781909 (+ 105us) server_negotiation.cc:407] Connection header received
0806 06:27:14.781973 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:27:14.781974 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:27:14.781989 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:27:14.782060 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:27:14.782415 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:27:14.782902 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:27:14.783565 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:27:14.783744 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:27:14.783791 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:27:14.786065 (+ 2274us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.12.130:43249: BlockingRecv error: recv got EOF from 127.21.12.130:43249 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
Aug 06 06:27:14 dist-test-slave-4tdj krb5kdc[931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.12.131@KRBTEST.COM for oryx/127.21.12.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 06:27:14 dist-test-slave-4tdj krb5kdc[931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.12.131@KRBTEST.COM for oryx/127.21.12.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 06:27:14.917841 2568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:27:14.913715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:46519 (local address 127.21.12.131:41763)
0806 06:27:14.913840 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0806 06:27:14.913851 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0806 06:27:14.913913 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:27:14.914038 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:27:14.914039 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:27:14.914209 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0806 06:27:14.914256 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:27:14.914258 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:27:14.914913 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:27:14.914914 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:27:14.915438 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:27:14.915440 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:27:14.915709 (+ 269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:27:14.915709 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:27:14.915748 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:27:14.916159 (+ 411us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 06:27:14.916164 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0806 06:27:14.917782 (+ 1618us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.12.190:46519: Server oryx/127.21.12.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":66,"threads_started":1}
W20260806 06:27:14.917981 2565 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 06:27:14.913806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.131:41763 (local address 127.21.12.190:46519)
0806 06:27:14.913853 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0806 06:27:14.913855 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:27:14.913896 (+ 41us) server_negotiation.cc:407] Connection header received
0806 06:27:14.913963 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:27:14.913964 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:27:14.913976 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:27:14.914020 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:27:14.914347 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:27:14.914840 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:27:14.915495 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:27:14.915664 (+ 169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:27:14.915705 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:27:14.917912 (+ 2207us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.12.131:41763: BlockingRecv error: recv got EOF from 127.21.12.131:41763 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
Aug 06 06:27:15 dist-test-slave-4tdj krb5kdc[931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.12.129@KRBTEST.COM for oryx/127.21.12.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 06:27:15 dist-test-slave-4tdj krb5kdc[931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.12.129@KRBTEST.COM for oryx/127.21.12.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 06:27:15.668833 2569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:27:15.661019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:46519 (local address 127.21.12.129:48023)
0806 06:27:15.661144 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0806 06:27:15.661156 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 06:27:15.661207 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:27:15.661351 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:27:15.661352 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:27:15.661526 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0806 06:27:15.661578 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:27:15.661580 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:27:15.662188 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:27:15.662189 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:27:15.662787 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:27:15.662790 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:27:15.663123 (+ 333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:27:15.663123 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:27:15.663161 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:27:15.666885 (+ 3724us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 06:27:15.666890 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0806 06:27:15.668755 (+ 1865us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.12.190:46519: Server oryx/127.21.12.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":70,"threads_started":1}
W20260806 06:27:15.668911 2570 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 06:27:15.661099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.129:48023 (local address 127.21.12.190:46519)
0806 06:27:15.661236 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0806 06:27:15.661238 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:27:15.661246 (+ 8us) server_negotiation.cc:407] Connection header received
0806 06:27:15.661268 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:27:15.661269 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:27:15.661280 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:27:15.661341 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:27:15.661633 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:27:15.662124 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:27:15.662871 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:27:15.663067 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:27:15.663120 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:27:15.668842 (+ 5722us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.12.129:48023: BlockingRecv error: recv got EOF from 127.21.12.129:48023 (error 108)
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":59,"threads_started":1}
Aug 06 06:27:15 dist-test-slave-4tdj krb5kdc[931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.12.130@KRBTEST.COM for oryx/127.21.12.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 06:27:15 dist-test-slave-4tdj krb5kdc[931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.21.12.130@KRBTEST.COM for oryx/127.21.12.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 06:27:15.791182 2573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:27:15.786600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:46519 (local address 127.21.12.130:36277)
0806 06:27:15.786738 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0806 06:27:15.786749 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0806 06:27:15.786804 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:27:15.786926 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:27:15.786927 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:27:15.787097 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0806 06:27:15.787147 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:27:15.787149 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:27:15.787780 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:27:15.787781 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:27:15.788321 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:27:15.788323 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:27:15.788651 (+ 328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:27:15.788652 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:27:15.788704 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:27:15.789190 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 06:27:15.789195 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0806 06:27:15.791082 (+ 1887us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.21.12.190:46519: Server oryx/127.21.12.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":63,"threads_started":1}
W20260806 06:27:15.791262 2570 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 06:27:15.786672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.130:36277 (local address 127.21.12.190:46519)
0806 06:27:15.786726 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0806 06:27:15.786728 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:27:15.786800 (+ 72us) server_negotiation.cc:407] Connection header received
0806 06:27:15.786862 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:27:15.786863 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:27:15.786874 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:27:15.786914 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:27:15.787206 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:27:15.787694 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:27:15.788387 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:27:15.788570 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:27:15.788614 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:27:15.791207 (+ 2593us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.21.12.130:36277: BlockingRecv error: recv got EOF from 127.21.12.130:36277 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
I20260806 06:27:15.845875 21554 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHhLhT/build/release/bin/kudu with pid 1830
I20260806 06:27:15.851197 21554 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHhLhT/build/release/bin/kudu with pid 2072
I20260806 06:27:15.862499 21554 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHhLhT/build/release/bin/kudu with pid 2276
I20260806 06:27:15.868916 21554 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHhLhT/build/release/bin/kudu with pid 1730
2026-08-06T06:27:15Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (18358 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-08-06T06:27:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T06:27:15Z Disabled control of system clock
I20260806 06:27:15.900138 21554 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHhLhT/build/release/bin/kudu
/tmp/dist-test-taskFHhLhT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785997581049705-21554-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785997581049705-21554-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785997581049705-21554-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785997581049705-21554-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.12.190:45467
--webserver_interface=127.21.12.190
--webserver_port=0
--builtin_ntp_servers=127.21.12.148:40871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.12.190:45467
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260806 06:27:15.978355 2578 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 06:27:15.978545 2578 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 06:27:15.978572 2578 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 06:27:15.979700 2578 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 06:27:15.979737 2578 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 06:27:15.979753 2578 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 06:27:15.979769 2578 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 06:27:15.979784 2578 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 06:27:15.981083 2578 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.148:40871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785997581049705-21554-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785997581049705-21554-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.12.190:45467
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.12.190:45467
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785997581049705-21554-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.12.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2578
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785997581049705-21554-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 05:11:02 UTC on ccycloud.quasar-hvzbbz.root.comops.site
build id 4569617
I20260806 06:27:15.981318 2578 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 06:27:15.981570 2578 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 06:27:15.986462 2590 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:27:15.986467 2587 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:27:15.986472 2586 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:27:15.986480 2578 server_base.cc:1034] running on GCE node
I20260806 06:27:15.986838 2578 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 06:27:15.987104 2578 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 06:27:15.988245 2578 hybrid_clock.cc:648] HybridClock initialized: now 1785997635988229 us; error 30 us; skew 500 ppm
I20260806 06:27:15.989421 2578 webserver.cc:460] Webserver started at http://127.21.12.190:45185/ using document root <none> and password file <none>
I20260806 06:27:15.989630 2578 fs_manager.cc:359] Metadata directory not provided
I20260806 06:27:15.989677 2578 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 06:27:15.989799 2578 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 06:27:15.990805 2578 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785997581049705-21554-0/minicluster-data/master-0/data/instance:
uuid: "187dd2e09d784ff2b2e57d1e351b1194"
format_stamp: "Formatted at 2026-08-06 06:27:15 on dist-test-slave-4tdj"
server_key: "e59b918c4d4a3c2485bfb05d5fbd3b5e"
server_key_iv: "f598268d92643d0726240e35441aa79a"
server_key_version: "encryptionkey@0"
I20260806 06:27:15.991117 2578 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785997581049705-21554-0/minicluster-data/master-0/wal/instance:
uuid: "187dd2e09d784ff2b2e57d1e351b1194"
format_stamp: "Formatted at 2026-08-06 06:27:15 on dist-test-slave-4tdj"
server_key: "e59b918c4d4a3c2485bfb05d5fbd3b5e"
server_key_iv: "f598268d92643d0726240e35441aa79a"
server_key_version: "encryptionkey@0"
I20260806 06:27:15.992281 2578 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260806 06:27:15.992904 2601 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 06:27:15.993067 2578 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260806 06:27:15.993153 2578 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785997581049705-21554-0/minicluster-data/master-0/data,/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785997581049705-21554-0/minicluster-data/master-0/wal
uuid: "187dd2e09d784ff2b2e57d1e351b1194"
format_stamp: "Formatted at 2026-08-06 06:27:15 on dist-test-slave-4tdj"
server_key: "e59b918c4d4a3c2485bfb05d5fbd3b5e"
server_key_iv: "f598268d92643d0726240e35441aa79a"
server_key_version: "encryptionkey@0"
I20260806 06:27:15.993218 2578 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785997581049705-21554-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785997581049705-21554-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785997581049705-21554-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 06:27:16.006918 2578 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 06:27:16.007169 2578 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 06:27:16.007287 2578 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 06:27:16.011055 2578 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.190:45467
I20260806 06:27:16.011101 2697 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.190:45467 every 8 connection(s)
I20260806 06:27:16.011399 2578 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785997581049705-21554-0/minicluster-data/master-0/data/info.pb
I20260806 06:27:16.012024 2698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 06:27:16.013991 21554 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHhLhT/build/release/bin/kudu as pid 2578
I20260806 06:27:16.014091 21554 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785997581049705-21554-0/minicluster-data/master-0/wal/instance
I20260806 06:27:16.014279 21554 external_mini_cluster.cc:1428] Setting key cfb1bba66760160eaf959a7775971174
I20260806 06:27:16.014832 2698 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 187dd2e09d784ff2b2e57d1e351b1194: Bootstrap starting.
I20260806 06:27:16.015411 2698 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 187dd2e09d784ff2b2e57d1e351b1194: Neither blocks nor log segments found. Creating new log.
I20260806 06:27:16.015717 2698 log.cc:824] T 00000000000000000000000000000000 P 187dd2e09d784ff2b2e57d1e351b1194: Log is configured to *not* fsync() on all Append() calls
I20260806 06:27:16.016355 2698 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 187dd2e09d784ff2b2e57d1e351b1194: No bootstrap required, opened a new log
I20260806 06:27:16.018441 2698 raft_consensus.cc:348] T 00000000000000000000000000000000 P 187dd2e09d784ff2b2e57d1e351b1194 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "187dd2e09d784ff2b2e57d1e351b1194" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 45467 } }
I20260806 06:27:16.018594 2698 raft_consensus.cc:374] T 00000000000000000000000000000000 P 187dd2e09d784ff2b2e57d1e351b1194 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 06:27:16.018620 2698 raft_consensus.cc:729] T 00000000000000000000000000000000 P 187dd2e09d784ff2b2e57d1e351b1194 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 187dd2e09d784ff2b2e57d1e351b1194, State: Initialized, Role: FOLLOWER
I20260806 06:27:16.018736 2698 consensus_queue.cc:260] T 00000000000000000000000000000000 P 187dd2e09d784ff2b2e57d1e351b1194 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "187dd2e09d784ff2b2e57d1e351b1194" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 45467 } }
I20260806 06:27:16.018827 2698 raft_consensus.cc:388] T 00000000000000000000000000000000 P 187dd2e09d784ff2b2e57d1e351b1194 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 06:27:16.018867 2698 raft_consensus.cc:482] T 00000000000000000000000000000000 P 187dd2e09d784ff2b2e57d1e351b1194 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 06:27:16.018941 2698 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 187dd2e09d784ff2b2e57d1e351b1194 [term 0 FOLLOWER]: Advancing to term 1
I20260806 06:27:16.019497 2698 raft_consensus.cc:504] T 00000000000000000000000000000000 P 187dd2e09d784ff2b2e57d1e351b1194 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "187dd2e09d784ff2b2e57d1e351b1194" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 45467 } }
I20260806 06:27:16.019609 2698 leader_election.cc:302] T 00000000000000000000000000000000 P 187dd2e09d784ff2b2e57d1e351b1194 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 187dd2e09d784ff2b2e57d1e351b1194; no voters:
I20260806 06:27:16.019711 2705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:27:16.014798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44390 (local address 127.21.12.190:45467)
0806 06:27:16.014972 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0806 06:27:16.014980 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 06:27:16.015003 (+ 23us) server_negotiation.cc:407] Connection header received
0806 06:27:16.015422 (+ 419us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:27:16.015428 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:27:16.015484 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:27:16.015619 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:27:16.015867 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:27:16.016745 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:27:16.017782 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:27:16.018165 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:27:16.018249 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:27:16.018921 (+ 672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:27:16.018926 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:27:16.018926 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 06:27:16.019128 (+ 202us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:27:16.019160 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 06:27:16.019170 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:27:16.019219 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0806 06:27:16.019438 (+ 219us) server_negotiation.cc:299] Negotiation successful
0806 06:27:16.019518 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":64,"threads_started":1}
I20260806 06:27:16.019760 2698 leader_election.cc:288] T 00000000000000000000000000000000 P 187dd2e09d784ff2b2e57d1e351b1194 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 06:27:16.019923 2712 sys_catalog.cc:455] T 00000000000000000000000000000000 P 187dd2e09d784ff2b2e57d1e351b1194 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "187dd2e09d784ff2b2e57d1e351b1194" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 45467 } } }
I20260806 06:27:16.019979 2698 sys_catalog.cc:564] T 00000000000000000000000000000000 P 187dd2e09d784ff2b2e57d1e351b1194 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 06:27:16.019997 2712 sys_catalog.cc:458] T 00000000000000000000000000000000 P 187dd2e09d784ff2b2e57d1e351b1194 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 06:27:16.020017 2711 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 187dd2e09d784ff2b2e57d1e351b1194 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 06:27:16.020103 2711 raft_consensus.cc:686] T 00000000000000000000000000000000 P 187dd2e09d784ff2b2e57d1e351b1194 [term 1 LEADER]: Becoming Leader. State: Replica: 187dd2e09d784ff2b2e57d1e351b1194, State: Running, Role: LEADER
I20260806 06:27:16.020169 2711 consensus_queue.cc:237] T 00000000000000000000000000000000 P 187dd2e09d784ff2b2e57d1e351b1194 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "187dd2e09d784ff2b2e57d1e351b1194" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 45467 } }
I20260806 06:27:16.020501 2712 sys_catalog.cc:455] T 00000000000000000000000000000000 P 187dd2e09d784ff2b2e57d1e351b1194 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 187dd2e09d784ff2b2e57d1e351b1194. Latest consensus state: current_term: 1 leader_uuid: "187dd2e09d784ff2b2e57d1e351b1194" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "187dd2e09d784ff2b2e57d1e351b1194" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 45467 } } }
I20260806 06:27:16.020576 2712 sys_catalog.cc:458] T 00000000000000000000000000000000 P 187dd2e09d784ff2b2e57d1e351b1194 [sys.catalog]: This master's current role is: LEADER
I20260806 06:27:16.020742 2720 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 06:27:16.021198 2720 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 06:27:16.022680 2720 catalog_manager.cc:1347] Generated new cluster ID: 8ca8a196ede34ec198a30f1c584ccbb1
I20260806 06:27:16.022724 2720 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 06:27:16.032305 2720 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 06:27:16.032948 2720 catalog_manager.cc:1504] Loading token signing keys...
I20260806 06:27:16.038025 2720 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 187dd2e09d784ff2b2e57d1e351b1194: Generated new TSK 0
I20260806 06:27:16.038182 2720 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 06:27:16.041113 21554 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHhLhT/build/release/bin/kudu
/tmp/dist-test-taskFHhLhT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785997581049705-21554-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785997581049705-21554-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785997581049705-21554-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785997581049705-21554-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.12.129:0
--local_ip_for_outbound_sockets=127.21.12.129
--webserver_interface=127.21.12.129
--webserver_port=0
--tserver_master_addrs=127.21.12.190:45467
--builtin_ntp_servers=127.21.12.148:40871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260806 06:27:16.123785 2738 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 06:27:16.123978 2738 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 06:27:16.124006 2738 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 06:27:16.125104 2738 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 06:27:16.125142 2738 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 06:27:16.125182 2738 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.12.129
I20260806 06:27:16.126576 2738 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.148:40871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785997581049705-21554-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785997581049705-21554-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.12.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785997581049705-21554-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.12.129
--webserver_port=0
--tserver_master_addrs=127.21.12.190:45467
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2738
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.12.129
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785997581049705-21554-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 05:11:02 UTC on ccycloud.quasar-hvzbbz.root.comops.site
build id 4569617
I20260806 06:27:16.126794 2738 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 06:27:16.127048 2738 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 06:27:16.132714 2748 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:27:16.132763 2738 server_base.cc:1034] running on GCE node
W20260806 06:27:16.132689 2751 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:27:16.132861 2747 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:27:16.133070 2738 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 06:27:16.133296 2738 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 06:27:16.134431 2738 hybrid_clock.cc:648] HybridClock initialized: now 1785997636134422 us; error 31 us; skew 500 ppm
I20260806 06:27:16.135623 2738 webserver.cc:460] Webserver started at http://127.21.12.129:37903/ using document root <none> and password file <none>
I20260806 06:27:16.135845 2738 fs_manager.cc:359] Metadata directory not provided
I20260806 06:27:16.135892 2738 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 06:27:16.136019 2738 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 06:27:16.137007 2738 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785997581049705-21554-0/minicluster-data/ts-0/data/instance:
uuid: "b9ee9bfda8ec46d881b1610c75775b87"
format_stamp: "Formatted at 2026-08-06 06:27:16 on dist-test-slave-4tdj"
server_key: "19105a3bf8ac7725fd87b13d5ae203d5"
server_key_iv: "854a8304b8be49df7ffa5e8cd366b588"
server_key_version: "encryptionkey@0"
I20260806 06:27:16.137328 2738 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785997581049705-21554-0/minicluster-data/ts-0/wal/instance:
uuid: "b9ee9bfda8ec46d881b1610c75775b87"
format_stamp: "Formatted at 2026-08-06 06:27:16 on dist-test-slave-4tdj"
server_key: "19105a3bf8ac7725fd87b13d5ae203d5"
server_key_iv: "854a8304b8be49df7ffa5e8cd366b588"
server_key_version: "encryptionkey@0"
I20260806 06:27:16.138571 2738 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260806 06:27:16.139325 2760 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 06:27:16.139478 2738 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260806 06:27:16.139564 2738 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785997581049705-21554-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785997581049705-21554-0/minicluster-data/ts-0/wal
uuid: "b9ee9bfda8ec46d881b1610c75775b87"
format_stamp: "Formatted at 2026-08-06 06:27:16 on dist-test-slave-4tdj"
server_key: "19105a3bf8ac7725fd87b13d5ae203d5"
server_key_iv: "854a8304b8be49df7ffa5e8cd366b588"
server_key_version: "encryptionkey@0"
I20260806 06:27:16.139631 2738 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785997581049705-21554-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785997581049705-21554-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785997581049705-21554-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 06:27:16.169740 2738 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 06:27:16.170042 2738 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 06:27:16.170168 2738 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 06:27:16.170413 2738 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 06:27:16.170732 2738 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 06:27:16.170768 2738 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 06:27:16.170796 2738 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 06:27:16.170819 2738 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 06:27:16.176071 2738 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.129:32945
I20260806 06:27:16.176148 2941 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.129:32945 every 8 connection(s)
I20260806 06:27:16.176419 2738 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785997581049705-21554-0/minicluster-data/ts-0/data/info.pb
I20260806 06:27:16.179364 21554 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHhLhT/build/release/bin/kudu as pid 2738
I20260806 06:27:16.179461 21554 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785997581049705-21554-0/minicluster-data/ts-0/wal/instance
I20260806 06:27:16.179643 21554 external_mini_cluster.cc:1428] Setting key 333a7011d2865d0fd7ad9b1770c829ff
I20260806 06:27:16.180584 21554 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHhLhT/build/release/bin/kudu
/tmp/dist-test-taskFHhLhT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785997581049705-21554-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785997581049705-21554-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785997581049705-21554-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785997581049705-21554-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.12.130:0
--local_ip_for_outbound_sockets=127.21.12.130
--webserver_interface=127.21.12.130
--webserver_port=0
--tserver_master_addrs=127.21.12.190:45467
--builtin_ntp_servers=127.21.12.148:40871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260806 06:27:16.181692 2705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:27:16.177085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.129:42093 (local address 127.21.12.190:45467)
0806 06:27:16.177136 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0806 06:27:16.177138 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:27:16.177390 (+ 252us) server_negotiation.cc:407] Connection header received
0806 06:27:16.177855 (+ 465us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:27:16.177856 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:27:16.177870 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:27:16.177920 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:27:16.178602 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:27:16.179112 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:27:16.180419 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:27:16.180607 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:27:16.180663 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:27:16.181474 (+ 811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:27:16.181475 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:27:16.181476 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 06:27:16.181480 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:27:16.181493 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 06:27:16.181496 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:27:16.181535 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0806 06:27:16.181612 (+ 77us) server_negotiation.cc:299] Negotiation successful
0806 06:27:16.181634 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260806 06:27:16.181921 2947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:27:16.177074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:45467 (local address 127.21.12.129:42093)
0806 06:27:16.177273 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0806 06:27:16.177306 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0806 06:27:16.177777 (+ 471us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:27:16.177972 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:27:16.177975 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:27:16.177983 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0806 06:27:16.178500 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:27:16.178506 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:27:16.179233 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:27:16.179235 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:27:16.180333 (+ 1098us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:27:16.180336 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:27:16.180690 (+ 354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:27:16.180691 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:27:16.180835 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:27:16.181334 (+ 499us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 06:27:16.181335 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0806 06:27:16.181371 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 06:27:16.181396 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 06:27:16.181546 (+ 150us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 06:27:16.181550 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 06:27:16.181553 (+ 3us) client_negotiation.cc:769] Sending connection context
0806 06:27:16.181612 (+ 59us) client_negotiation.cc:240] Negotiation successful
0806 06:27:16.181713 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":97,"threads_started":1}
I20260806 06:27:16.182514 2943 heartbeater.cc:347] Connected to a master server at 127.21.12.190:45467
I20260806 06:27:16.182614 2943 heartbeater.cc:464] Registering TS with master...
I20260806 06:27:16.182802 2943 heartbeater.cc:511] Master 127.21.12.190:45467 requested a full tablet report, sending...
I20260806 06:27:16.183312 2639 ts_manager.cc:194] Registered new tserver with Master: b9ee9bfda8ec46d881b1610c75775b87 (127.21.12.129:32945)
I20260806 06:27:16.183923 2639 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.12.129:42093
W20260806 06:27:16.260896 2949 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 06:27:16.261102 2949 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 06:27:16.261132 2949 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 06:27:16.262247 2949 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 06:27:16.262284 2949 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 06:27:16.262346 2949 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.12.130
I20260806 06:27:16.263667 2949 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.148:40871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785997581049705-21554-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785997581049705-21554-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.12.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785997581049705-21554-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.12.130
--webserver_port=0
--tserver_master_addrs=127.21.12.190:45467
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2949
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.12.130
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785997581049705-21554-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 05:11:02 UTC on ccycloud.quasar-hvzbbz.root.comops.site
build id 4569617
I20260806 06:27:16.263899 2949 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 06:27:16.264178 2949 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 06:27:16.268940 2954 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:27:16.269037 2959 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:27:16.268999 2957 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:27:16.269130 2949 server_base.cc:1034] running on GCE node
I20260806 06:27:16.269332 2949 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 06:27:16.269546 2949 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 06:27:16.270686 2949 hybrid_clock.cc:648] HybridClock initialized: now 1785997636270673 us; error 28 us; skew 500 ppm
I20260806 06:27:16.271854 2949 webserver.cc:460] Webserver started at http://127.21.12.130:40403/ using document root <none> and password file <none>
I20260806 06:27:16.272075 2949 fs_manager.cc:359] Metadata directory not provided
I20260806 06:27:16.272121 2949 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 06:27:16.272244 2949 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 06:27:16.273233 2949 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785997581049705-21554-0/minicluster-data/ts-1/data/instance:
uuid: "dbd869531f264f38a6461e83ac675153"
format_stamp: "Formatted at 2026-08-06 06:27:16 on dist-test-slave-4tdj"
server_key: "f2b1b06819ed99addd033bd64f552f12"
server_key_iv: "fce2bda378f382960843fd92a8cda19d"
server_key_version: "encryptionkey@0"
I20260806 06:27:16.273546 2949 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785997581049705-21554-0/minicluster-data/ts-1/wal/instance:
uuid: "dbd869531f264f38a6461e83ac675153"
format_stamp: "Formatted at 2026-08-06 06:27:16 on dist-test-slave-4tdj"
server_key: "f2b1b06819ed99addd033bd64f552f12"
server_key_iv: "fce2bda378f382960843fd92a8cda19d"
server_key_version: "encryptionkey@0"
I20260806 06:27:16.274780 2949 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260806 06:27:16.275439 2967 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 06:27:16.275589 2949 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260806 06:27:16.275671 2949 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785997581049705-21554-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785997581049705-21554-0/minicluster-data/ts-1/wal
uuid: "dbd869531f264f38a6461e83ac675153"
format_stamp: "Formatted at 2026-08-06 06:27:16 on dist-test-slave-4tdj"
server_key: "f2b1b06819ed99addd033bd64f552f12"
server_key_iv: "fce2bda378f382960843fd92a8cda19d"
server_key_version: "encryptionkey@0"
I20260806 06:27:16.275758 2949 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785997581049705-21554-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785997581049705-21554-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785997581049705-21554-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 06:27:16.299631 2949 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 06:27:16.299909 2949 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 06:27:16.300010 2949 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 06:27:16.300194 2949 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 06:27:16.300465 2949 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 06:27:16.300494 2949 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 06:27:16.300519 2949 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 06:27:16.300535 2949 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 06:27:16.305960 2949 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.130:39719
I20260806 06:27:16.306030 3146 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.130:39719 every 8 connection(s)
I20260806 06:27:16.306334 2949 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785997581049705-21554-0/minicluster-data/ts-1/data/info.pb
I20260806 06:27:16.311203 2705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:27:16.307034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.130:55051 (local address 127.21.12.190:45467)
0806 06:27:16.307086 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0806 06:27:16.307089 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 06:27:16.307312 (+ 223us) server_negotiation.cc:407] Connection header received
0806 06:27:16.307785 (+ 473us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:27:16.307786 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:27:16.307797 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:27:16.307854 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:27:16.308464 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:27:16.308949 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:27:16.309972 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:27:16.310169 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:27:16.310230 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:27:16.310982 (+ 752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:27:16.310983 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:27:16.310984 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 06:27:16.310988 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:27:16.311002 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 06:27:16.311006 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:27:16.311071 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0806 06:27:16.311125 (+ 54us) server_negotiation.cc:299] Negotiation successful
0806 06:27:16.311147 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260806 06:27:16.311373 3152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:27:16.307008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:45467 (local address 127.21.12.130:55051)
0806 06:27:16.307188 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0806 06:27:16.307229 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0806 06:27:16.307700 (+ 471us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:27:16.307909 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:27:16.307912 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:27:16.307921 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0806 06:27:16.308389 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:27:16.308393 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:27:16.309011 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:27:16.309011 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:27:16.309869 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:27:16.309873 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:27:16.310240 (+ 367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:27:16.310241 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:27:16.310365 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:27:16.310841 (+ 476us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 06:27:16.310842 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0806 06:27:16.310869 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 06:27:16.310890 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 06:27:16.311074 (+ 184us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 06:27:16.311076 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 06:27:16.311079 (+ 3us) client_negotiation.cc:769] Sending connection context
0806 06:27:16.311119 (+ 40us) client_negotiation.cc:240] Negotiation successful
0806 06:27:16.311193 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":66,"threads_started":1}
I20260806 06:27:16.311805 3148 heartbeater.cc:347] Connected to a master server at 127.21.12.190:45467
I20260806 06:27:16.311897 3148 heartbeater.cc:464] Registering TS with master...
I20260806 06:27:16.312075 3148 heartbeater.cc:511] Master 127.21.12.190:45467 requested a full tablet report, sending...
I20260806 06:27:16.312420 2639 ts_manager.cc:194] Registered new tserver with Master: dbd869531f264f38a6461e83ac675153 (127.21.12.130:39719)
I20260806 06:27:16.312783 2639 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.12.130:55051
I20260806 06:27:16.316346 21554 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHhLhT/build/release/bin/kudu as pid 2949
I20260806 06:27:16.316423 21554 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785997581049705-21554-0/minicluster-data/ts-1/wal/instance
I20260806 06:27:16.316582 21554 external_mini_cluster.cc:1428] Setting key d89b9a4233c7b387f72911fc657f0538
I20260806 06:27:16.317441 21554 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHhLhT/build/release/bin/kudu
/tmp/dist-test-taskFHhLhT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785997581049705-21554-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785997581049705-21554-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785997581049705-21554-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785997581049705-21554-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.12.131:0
--local_ip_for_outbound_sockets=127.21.12.131
--webserver_interface=127.21.12.131
--webserver_port=0
--tserver_master_addrs=127.21.12.190:45467
--builtin_ntp_servers=127.21.12.148:40871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260806 06:27:16.402022 3155 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 06:27:16.402321 3155 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 06:27:16.402385 3155 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 06:27:16.403539 3155 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 06:27:16.403582 3155 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 06:27:16.403694 3155 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.12.131
I20260806 06:27:16.405221 3155 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.148:40871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785997581049705-21554-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785997581049705-21554-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.12.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785997581049705-21554-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.12.131
--webserver_port=0
--tserver_master_addrs=127.21.12.190:45467
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3155
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.12.131
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785997581049705-21554-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 05:11:02 UTC on ccycloud.quasar-hvzbbz.root.comops.site
build id 4569617
I20260806 06:27:16.405457 3155 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 06:27:16.405720 3155 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 06:27:16.410362 3166 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:27:16.410352 3164 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:27:16.410429 3155 server_base.cc:1034] running on GCE node
W20260806 06:27:16.410341 3169 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:27:16.410763 3155 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 06:27:16.411005 3155 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 06:27:16.412323 3155 hybrid_clock.cc:648] HybridClock initialized: now 1785997636412290 us; error 43 us; skew 500 ppm
I20260806 06:27:16.413547 3155 webserver.cc:460] Webserver started at http://127.21.12.131:45525/ using document root <none> and password file <none>
I20260806 06:27:16.413760 3155 fs_manager.cc:359] Metadata directory not provided
I20260806 06:27:16.413805 3155 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 06:27:16.413931 3155 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 06:27:16.414991 3155 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785997581049705-21554-0/minicluster-data/ts-2/data/instance:
uuid: "730e444f426d4092b20d5ad6e4389994"
format_stamp: "Formatted at 2026-08-06 06:27:16 on dist-test-slave-4tdj"
server_key: "386244d80422565c79b2b9fbe1a3231e"
server_key_iv: "240c6d68c51bfa365a89cdf03812bbcb"
server_key_version: "encryptionkey@0"
I20260806 06:27:16.415314 3155 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785997581049705-21554-0/minicluster-data/ts-2/wal/instance:
uuid: "730e444f426d4092b20d5ad6e4389994"
format_stamp: "Formatted at 2026-08-06 06:27:16 on dist-test-slave-4tdj"
server_key: "386244d80422565c79b2b9fbe1a3231e"
server_key_iv: "240c6d68c51bfa365a89cdf03812bbcb"
server_key_version: "encryptionkey@0"
I20260806 06:27:16.416467 3155 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260806 06:27:16.417078 3175 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 06:27:16.417228 3155 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260806 06:27:16.417309 3155 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785997581049705-21554-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785997581049705-21554-0/minicluster-data/ts-2/wal
uuid: "730e444f426d4092b20d5ad6e4389994"
format_stamp: "Formatted at 2026-08-06 06:27:16 on dist-test-slave-4tdj"
server_key: "386244d80422565c79b2b9fbe1a3231e"
server_key_iv: "240c6d68c51bfa365a89cdf03812bbcb"
server_key_version: "encryptionkey@0"
I20260806 06:27:16.417367 3155 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785997581049705-21554-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785997581049705-21554-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785997581049705-21554-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 06:27:16.428197 3155 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 06:27:16.428431 3155 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 06:27:16.428547 3155 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 06:27:16.428723 3155 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 06:27:16.429008 3155 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 06:27:16.429042 3155 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 06:27:16.429067 3155 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 06:27:16.429093 3155 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 06:27:16.434448 3155 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.131:42739
I20260806 06:27:16.434508 3361 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.131:42739 every 8 connection(s)
I20260806 06:27:16.434772 3155 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785997581049705-21554-0/minicluster-data/ts-2/data/info.pb
I20260806 06:27:16.439169 2705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:27:16.435365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.131:38557 (local address 127.21.12.190:45467)
0806 06:27:16.435415 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0806 06:27:16.435417 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:27:16.435618 (+ 201us) server_negotiation.cc:407] Connection header received
0806 06:27:16.436076 (+ 458us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:27:16.436077 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:27:16.436089 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:27:16.436157 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:27:16.436738 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:27:16.437207 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:27:16.438068 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:27:16.438258 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:27:16.438341 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:27:16.438969 (+ 628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:27:16.438971 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:27:16.438971 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 06:27:16.438975 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:27:16.438988 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 06:27:16.438991 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:27:16.439040 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0806 06:27:16.439093 (+ 53us) server_negotiation.cc:299] Negotiation successful
0806 06:27:16.439114 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260806 06:27:16.439355 3368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:27:16.435336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:45467 (local address 127.21.12.131:38557)
0806 06:27:16.435509 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0806 06:27:16.435546 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0806 06:27:16.435975 (+ 429us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:27:16.436213 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:27:16.436217 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:27:16.436225 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0806 06:27:16.436676 (+ 451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:27:16.436681 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:27:16.437263 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:27:16.437264 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:27:16.437995 (+ 731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:27:16.437996 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:27:16.438341 (+ 345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:27:16.438342 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:27:16.438459 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:27:16.438858 (+ 399us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 06:27:16.438858 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0806 06:27:16.438883 (+ 25us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 06:27:16.438903 (+ 20us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 06:27:16.439043 (+ 140us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 06:27:16.439045 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 06:27:16.439047 (+ 2us) client_negotiation.cc:769] Sending connection context
0806 06:27:16.439089 (+ 42us) client_negotiation.cc:240] Negotiation successful
0806 06:27:16.439160 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":81,"threads_started":1}
I20260806 06:27:16.439764 3365 heartbeater.cc:347] Connected to a master server at 127.21.12.190:45467
I20260806 06:27:16.439842 3365 heartbeater.cc:464] Registering TS with master...
I20260806 06:27:16.440004 3365 heartbeater.cc:511] Master 127.21.12.190:45467 requested a full tablet report, sending...
I20260806 06:27:16.440307 2639 ts_manager.cc:194] Registered new tserver with Master: 730e444f426d4092b20d5ad6e4389994 (127.21.12.131:42739)
I20260806 06:27:16.440659 2639 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.12.131:38557
I20260806 06:27:16.442534 21554 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHhLhT/build/release/bin/kudu as pid 3155
I20260806 06:27:16.442612 21554 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785997581049705-21554-0/minicluster-data/ts-2/wal/instance
I20260806 06:27:16.442803 21554 external_mini_cluster.cc:1428] Setting key 12486ef22e087c76539893d1cb890934
I20260806 06:27:16.443512 21554 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260806 06:27:16.445123 2705 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 06:27:16.444711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44398 (local address 127.21.12.190:45467)
0806 06:27:16.444762 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0806 06:27:16.444764 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:27:16.444864 (+ 100us) server_negotiation.cc:407] Connection header received
0806 06:27:16.444910 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:27:16.444911 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:27:16.444919 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:27:16.444954 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:27:16.445066 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44398: BlockingRecv error: recv got EOF from 127.0.0.1:44398 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
I20260806 06:27:16.445178 21554 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHhLhT/build/release/bin/kudu with pid 2738
I20260806 06:27:16.465790 21554 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHhLhT/build/release/bin/kudu with pid 2949
I20260806 06:27:16.472527 21554 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHhLhT/build/release/bin/kudu with pid 3155
I20260806 06:27:16.482182 21554 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHhLhT/build/release/bin/kudu with pid 2578
2026-08-06T06:27:16Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (609 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-08-06T06:27:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T06:27:16Z Disabled control of system clock
I20260806 06:27:16.509802 21554 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHhLhT/build/release/bin/kudu
/tmp/dist-test-taskFHhLhT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785997581049705-21554-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785997581049705-21554-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785997581049705-21554-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785997581049705-21554-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.12.190:36197
--webserver_interface=127.21.12.190
--webserver_port=0
--builtin_ntp_servers=127.21.12.148:34657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.12.190:36197
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260806 06:27:16.589239 3382 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 06:27:16.589432 3382 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 06:27:16.589452 3382 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 06:27:16.590539 3382 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 06:27:16.590574 3382 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 06:27:16.590584 3382 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 06:27:16.590595 3382 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 06:27:16.590610 3382 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 06:27:16.591882 3382 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.148:34657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785997581049705-21554-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785997581049705-21554-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.12.190:36197
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.12.190:36197
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785997581049705-21554-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.12.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3382
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785997581049705-21554-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 05:11:02 UTC on ccycloud.quasar-hvzbbz.root.comops.site
build id 4569617
I20260806 06:27:16.592069 3382 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 06:27:16.592295 3382 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 06:27:16.596647 3390 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:27:16.596683 3391 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:27:16.596698 3394 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:27:16.596983 3382 server_base.cc:1034] running on GCE node
I20260806 06:27:16.597188 3382 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 06:27:16.597419 3382 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 06:27:16.598577 3382 hybrid_clock.cc:648] HybridClock initialized: now 1785997636598564 us; error 27 us; skew 500 ppm
I20260806 06:27:16.599727 3382 webserver.cc:460] Webserver started at http://127.21.12.190:39513/ using document root <none> and password file <none>
I20260806 06:27:16.599917 3382 fs_manager.cc:359] Metadata directory not provided
I20260806 06:27:16.599956 3382 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 06:27:16.600055 3382 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 06:27:16.600971 3382 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785997581049705-21554-0/minicluster-data/master-0/data/instance:
uuid: "9c8fc19f2304449fa6acb56def9865f6"
format_stamp: "Formatted at 2026-08-06 06:27:16 on dist-test-slave-4tdj"
server_key: "4cdfa3b56a58170b2280cb6f7deb2ae5"
server_key_iv: "3ade1d247de9d804bd5a8e9f28ea0805"
server_key_version: "encryptionkey@0"
I20260806 06:27:16.601254 3382 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785997581049705-21554-0/minicluster-data/master-0/wal/instance:
uuid: "9c8fc19f2304449fa6acb56def9865f6"
format_stamp: "Formatted at 2026-08-06 06:27:16 on dist-test-slave-4tdj"
server_key: "4cdfa3b56a58170b2280cb6f7deb2ae5"
server_key_iv: "3ade1d247de9d804bd5a8e9f28ea0805"
server_key_version: "encryptionkey@0"
I20260806 06:27:16.602384 3382 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260806 06:27:16.603060 3403 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 06:27:16.603284 3382 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260806 06:27:16.603368 3382 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785997581049705-21554-0/minicluster-data/master-0/data,/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785997581049705-21554-0/minicluster-data/master-0/wal
uuid: "9c8fc19f2304449fa6acb56def9865f6"
format_stamp: "Formatted at 2026-08-06 06:27:16 on dist-test-slave-4tdj"
server_key: "4cdfa3b56a58170b2280cb6f7deb2ae5"
server_key_iv: "3ade1d247de9d804bd5a8e9f28ea0805"
server_key_version: "encryptionkey@0"
I20260806 06:27:16.603436 3382 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785997581049705-21554-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785997581049705-21554-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785997581049705-21554-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 06:27:16.606591 3382 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 06:27:16.606818 3382 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 06:27:16.606930 3382 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 06:27:16.610733 3382 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.190:36197
I20260806 06:27:16.610745 3480 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.190:36197 every 8 connection(s)
I20260806 06:27:16.611059 3382 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785997581049705-21554-0/minicluster-data/master-0/data/info.pb
I20260806 06:27:16.611644 3481 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 06:27:16.613406 21554 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHhLhT/build/release/bin/kudu as pid 3382
I20260806 06:27:16.613477 21554 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785997581049705-21554-0/minicluster-data/master-0/wal/instance
I20260806 06:27:16.613634 21554 external_mini_cluster.cc:1428] Setting key 66f5899f40723d2108aae14557c100cf
I20260806 06:27:16.614360 3481 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c8fc19f2304449fa6acb56def9865f6: Bootstrap starting.
I20260806 06:27:16.615016 3481 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9c8fc19f2304449fa6acb56def9865f6: Neither blocks nor log segments found. Creating new log.
I20260806 06:27:16.615325 3481 log.cc:824] T 00000000000000000000000000000000 P 9c8fc19f2304449fa6acb56def9865f6: Log is configured to *not* fsync() on all Append() calls
I20260806 06:27:16.616046 3486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:27:16.614098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33358 (local address 127.21.12.190:36197)
0806 06:27:16.614325 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0806 06:27:16.614333 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 06:27:16.614364 (+ 31us) server_negotiation.cc:407] Connection header received
0806 06:27:16.614882 (+ 518us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:27:16.614890 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:27:16.614941 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:27:16.615084 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:27:16.615623 (+ 539us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:27:16.615626 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:27:16.615626 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 06:27:16.615631 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:27:16.615685 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 06:27:16.615700 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:27:16.615795 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0806 06:27:16.615825 (+ 30us) server_negotiation.cc:299] Negotiation successful
0806 06:27:16.615881 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":104,"threads_started":1}
I20260806 06:27:16.616076 3481 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c8fc19f2304449fa6acb56def9865f6: No bootstrap required, opened a new log
I20260806 06:27:16.617841 3481 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9c8fc19f2304449fa6acb56def9865f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c8fc19f2304449fa6acb56def9865f6" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 36197 } }
I20260806 06:27:16.617967 3481 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9c8fc19f2304449fa6acb56def9865f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 06:27:16.617988 3481 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9c8fc19f2304449fa6acb56def9865f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c8fc19f2304449fa6acb56def9865f6, State: Initialized, Role: FOLLOWER
I20260806 06:27:16.618090 3481 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9c8fc19f2304449fa6acb56def9865f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c8fc19f2304449fa6acb56def9865f6" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 36197 } }
I20260806 06:27:16.618161 3481 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9c8fc19f2304449fa6acb56def9865f6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 06:27:16.618182 3481 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9c8fc19f2304449fa6acb56def9865f6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 06:27:16.618219 3481 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9c8fc19f2304449fa6acb56def9865f6 [term 0 FOLLOWER]: Advancing to term 1
I20260806 06:27:16.618722 3481 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9c8fc19f2304449fa6acb56def9865f6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c8fc19f2304449fa6acb56def9865f6" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 36197 } }
I20260806 06:27:16.618811 3481 leader_election.cc:302] T 00000000000000000000000000000000 P 9c8fc19f2304449fa6acb56def9865f6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9c8fc19f2304449fa6acb56def9865f6; no voters:
I20260806 06:27:16.618935 3481 leader_election.cc:288] T 00000000000000000000000000000000 P 9c8fc19f2304449fa6acb56def9865f6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 06:27:16.618974 3488 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9c8fc19f2304449fa6acb56def9865f6 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 06:27:16.619127 3488 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9c8fc19f2304449fa6acb56def9865f6 [term 1 LEADER]: Becoming Leader. State: Replica: 9c8fc19f2304449fa6acb56def9865f6, State: Running, Role: LEADER
I20260806 06:27:16.619143 3481 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9c8fc19f2304449fa6acb56def9865f6 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 06:27:16.619223 3488 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9c8fc19f2304449fa6acb56def9865f6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c8fc19f2304449fa6acb56def9865f6" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 36197 } }
I20260806 06:27:16.619571 3491 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c8fc19f2304449fa6acb56def9865f6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9c8fc19f2304449fa6acb56def9865f6. Latest consensus state: current_term: 1 leader_uuid: "9c8fc19f2304449fa6acb56def9865f6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c8fc19f2304449fa6acb56def9865f6" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 36197 } } }
I20260806 06:27:16.619692 3491 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c8fc19f2304449fa6acb56def9865f6 [sys.catalog]: This master's current role is: LEADER
I20260806 06:27:16.619566 3490 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c8fc19f2304449fa6acb56def9865f6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9c8fc19f2304449fa6acb56def9865f6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c8fc19f2304449fa6acb56def9865f6" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 36197 } } }
I20260806 06:27:16.619978 3490 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c8fc19f2304449fa6acb56def9865f6 [sys.catalog]: This master's current role is: LEADER
I20260806 06:27:16.619892 3499 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 06:27:16.620460 3499 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 06:27:16.621855 3499 catalog_manager.cc:1347] Generated new cluster ID: b0648c9d020d4c4296c656d41ac17317
I20260806 06:27:16.621899 3499 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 06:27:16.632535 3499 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 06:27:16.632678 3499 catalog_manager.cc:1504] Loading token signing keys...
I20260806 06:27:16.636245 3499 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9c8fc19f2304449fa6acb56def9865f6: Generated new TSK 0
I20260806 06:27:16.636396 3499 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 06:27:16.640405 21554 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHhLhT/build/release/bin/kudu
/tmp/dist-test-taskFHhLhT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785997581049705-21554-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785997581049705-21554-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785997581049705-21554-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785997581049705-21554-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.12.129:0
--local_ip_for_outbound_sockets=127.21.12.129
--webserver_interface=127.21.12.129
--webserver_port=0
--tserver_master_addrs=127.21.12.190:36197
--builtin_ntp_servers=127.21.12.148:34657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260806 06:27:16.736404 3514 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 06:27:16.736589 3514 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 06:27:16.736609 3514 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 06:27:16.737725 3514 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 06:27:16.737762 3514 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 06:27:16.737785 3514 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.12.129
I20260806 06:27:16.739153 3514 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.148:34657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785997581049705-21554-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785997581049705-21554-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.12.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785997581049705-21554-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.12.129
--webserver_port=0
--tserver_master_addrs=127.21.12.190:36197
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3514
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.12.129
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785997581049705-21554-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 05:11:02 UTC on ccycloud.quasar-hvzbbz.root.comops.site
build id 4569617
I20260806 06:27:16.739369 3514 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 06:27:16.739634 3514 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 06:27:16.744429 3523 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:27:16.744493 3529 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:27:16.744518 3524 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:27:16.745052 3514 server_base.cc:1034] running on GCE node
I20260806 06:27:16.745258 3514 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 06:27:16.745469 3514 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 06:27:16.746610 3514 hybrid_clock.cc:648] HybridClock initialized: now 1785997636746600 us; error 36 us; skew 500 ppm
I20260806 06:27:16.747749 3514 webserver.cc:460] Webserver started at http://127.21.12.129:44519/ using document root <none> and password file <none>
I20260806 06:27:16.747970 3514 fs_manager.cc:359] Metadata directory not provided
I20260806 06:27:16.748013 3514 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 06:27:16.748134 3514 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 06:27:16.749126 3514 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785997581049705-21554-0/minicluster-data/ts-0/data/instance:
uuid: "63ea0c8b23ea4c1a9322046b57b87848"
format_stamp: "Formatted at 2026-08-06 06:27:16 on dist-test-slave-4tdj"
server_key: "3799156881374ce26d2dfba3181c9ac9"
server_key_iv: "490491028fb5b19d6cd6dff9f459bda3"
server_key_version: "encryptionkey@0"
I20260806 06:27:16.749440 3514 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785997581049705-21554-0/minicluster-data/ts-0/wal/instance:
uuid: "63ea0c8b23ea4c1a9322046b57b87848"
format_stamp: "Formatted at 2026-08-06 06:27:16 on dist-test-slave-4tdj"
server_key: "3799156881374ce26d2dfba3181c9ac9"
server_key_iv: "490491028fb5b19d6cd6dff9f459bda3"
server_key_version: "encryptionkey@0"
I20260806 06:27:16.750643 3514 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260806 06:27:16.751319 3536 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 06:27:16.751466 3514 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260806 06:27:16.751543 3514 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785997581049705-21554-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785997581049705-21554-0/minicluster-data/ts-0/wal
uuid: "63ea0c8b23ea4c1a9322046b57b87848"
format_stamp: "Formatted at 2026-08-06 06:27:16 on dist-test-slave-4tdj"
server_key: "3799156881374ce26d2dfba3181c9ac9"
server_key_iv: "490491028fb5b19d6cd6dff9f459bda3"
server_key_version: "encryptionkey@0"
I20260806 06:27:16.751610 3514 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785997581049705-21554-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785997581049705-21554-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785997581049705-21554-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 06:27:16.754724 3514 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 06:27:16.754958 3514 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 06:27:16.755071 3514 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 06:27:16.755252 3514 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 06:27:16.755548 3514 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 06:27:16.755586 3514 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 06:27:16.755611 3514 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 06:27:16.755635 3514 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 06:27:16.761780 3514 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.129:39407
I20260806 06:27:16.761832 3715 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.129:39407 every 8 connection(s)
I20260806 06:27:16.762117 3514 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785997581049705-21554-0/minicluster-data/ts-0/data/info.pb
I20260806 06:27:16.764719 3486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:27:16.762775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.129:52479 (local address 127.21.12.190:36197)
0806 06:27:16.762846 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0806 06:27:16.762848 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:27:16.763022 (+ 174us) server_negotiation.cc:407] Connection header received
0806 06:27:16.763443 (+ 421us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:27:16.763445 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:27:16.763449 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:27:16.763494 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:27:16.764557 (+ 1063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:27:16.764558 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:27:16.764558 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 06:27:16.764560 (+ 2us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:27:16.764570 (+ 10us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 06:27:16.764573 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:27:16.764604 (+ 31us) server_negotiation.cc:1035] Waiting for connection context
0806 06:27:16.764650 (+ 46us) server_negotiation.cc:299] Negotiation successful
0806 06:27:16.764660 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260806 06:27:16.764901 3719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:27:16.762742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:36197 (local address 127.21.12.129:52479)
0806 06:27:16.762917 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0806 06:27:16.762949 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0806 06:27:16.763372 (+ 423us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:27:16.763547 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:27:16.763550 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:27:16.763558 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0806 06:27:16.764433 (+ 875us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 06:27:16.764434 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0806 06:27:16.764461 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 06:27:16.764489 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 06:27:16.764605 (+ 116us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 06:27:16.764607 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 06:27:16.764609 (+ 2us) client_negotiation.cc:769] Sending connection context
0806 06:27:16.764655 (+ 46us) client_negotiation.cc:240] Negotiation successful
0806 06:27:16.764709 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":76,"threads_started":1}
I20260806 06:27:16.765362 3716 heartbeater.cc:347] Connected to a master server at 127.21.12.190:36197
I20260806 06:27:16.765447 3716 heartbeater.cc:464] Registering TS with master...
I20260806 06:27:16.765586 3716 heartbeater.cc:511] Master 127.21.12.190:36197 requested a full tablet report, sending...
I20260806 06:27:16.766053 3429 ts_manager.cc:194] Registered new tserver with Master: 63ea0c8b23ea4c1a9322046b57b87848 (127.21.12.129:39407)
I20260806 06:27:16.766119 21554 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHhLhT/build/release/bin/kudu as pid 3514
I20260806 06:27:16.766248 21554 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785997581049705-21554-0/minicluster-data/ts-0/wal/instance
I20260806 06:27:16.766495 21554 external_mini_cluster.cc:1428] Setting key 1db33f42ab1d66c84707d1893236b0e3
I20260806 06:27:16.767290 21554 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHhLhT/build/release/bin/kudu
/tmp/dist-test-taskFHhLhT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785997581049705-21554-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785997581049705-21554-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785997581049705-21554-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785997581049705-21554-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.12.130:0
--local_ip_for_outbound_sockets=127.21.12.130
--webserver_interface=127.21.12.130
--webserver_port=0
--tserver_master_addrs=127.21.12.190:36197
--builtin_ntp_servers=127.21.12.148:34657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260806 06:27:16.845574 3721 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 06:27:16.845752 3721 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 06:27:16.845772 3721 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 06:27:16.846897 3721 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 06:27:16.846935 3721 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 06:27:16.846961 3721 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.12.130
I20260806 06:27:16.848260 3721 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.148:34657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785997581049705-21554-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785997581049705-21554-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.12.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785997581049705-21554-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.12.130
--webserver_port=0
--tserver_master_addrs=127.21.12.190:36197
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3721
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.12.130
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785997581049705-21554-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 05:11:02 UTC on ccycloud.quasar-hvzbbz.root.comops.site
build id 4569617
I20260806 06:27:16.848452 3721 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 06:27:16.848697 3721 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 06:27:16.853220 3736 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:27:16.853256 3734 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:27:16.853193 3729 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:27:16.853365 3721 server_base.cc:1034] running on GCE node
I20260806 06:27:16.853565 3721 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 06:27:16.853808 3721 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 06:27:16.854961 3721 hybrid_clock.cc:648] HybridClock initialized: now 1785997636854952 us; error 28 us; skew 500 ppm
I20260806 06:27:16.856063 3721 webserver.cc:460] Webserver started at http://127.21.12.130:34753/ using document root <none> and password file <none>
I20260806 06:27:16.856266 3721 fs_manager.cc:359] Metadata directory not provided
I20260806 06:27:16.856309 3721 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 06:27:16.856427 3721 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 06:27:16.857440 3721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785997581049705-21554-0/minicluster-data/ts-1/data/instance:
uuid: "0e1012eecb414738b0d96589e5b48054"
format_stamp: "Formatted at 2026-08-06 06:27:16 on dist-test-slave-4tdj"
server_key: "aa3c40d9595a53855a6aac37b4ce8533"
server_key_iv: "d472b56e7454a9cf6ad1f2b66aa3a9f7"
server_key_version: "encryptionkey@0"
I20260806 06:27:16.857758 3721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785997581049705-21554-0/minicluster-data/ts-1/wal/instance:
uuid: "0e1012eecb414738b0d96589e5b48054"
format_stamp: "Formatted at 2026-08-06 06:27:16 on dist-test-slave-4tdj"
server_key: "aa3c40d9595a53855a6aac37b4ce8533"
server_key_iv: "d472b56e7454a9cf6ad1f2b66aa3a9f7"
server_key_version: "encryptionkey@0"
I20260806 06:27:16.858980 3721 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260806 06:27:16.859628 3742 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 06:27:16.859784 3721 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260806 06:27:16.859871 3721 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785997581049705-21554-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785997581049705-21554-0/minicluster-data/ts-1/wal
uuid: "0e1012eecb414738b0d96589e5b48054"
format_stamp: "Formatted at 2026-08-06 06:27:16 on dist-test-slave-4tdj"
server_key: "aa3c40d9595a53855a6aac37b4ce8533"
server_key_iv: "d472b56e7454a9cf6ad1f2b66aa3a9f7"
server_key_version: "encryptionkey@0"
I20260806 06:27:16.859925 3721 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785997581049705-21554-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785997581049705-21554-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785997581049705-21554-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 06:27:16.863435 3721 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 06:27:16.863644 3721 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 06:27:16.863732 3721 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 06:27:16.863871 3721 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 06:27:16.864120 3721 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 06:27:16.864149 3721 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 06:27:16.864167 3721 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 06:27:16.864183 3721 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 06:27:16.869427 3721 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.130:42575
I20260806 06:27:16.869491 3908 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.130:42575 every 8 connection(s)
I20260806 06:27:16.869753 3721 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785997581049705-21554-0/minicluster-data/ts-1/data/info.pb
I20260806 06:27:16.871840 21554 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHhLhT/build/release/bin/kudu as pid 3721
I20260806 06:27:16.871929 21554 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785997581049705-21554-0/minicluster-data/ts-1/wal/instance
I20260806 06:27:16.872104 21554 external_mini_cluster.cc:1428] Setting key 80166af3737079af7040861d9ee4af19
I20260806 06:27:16.872611 3486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:27:16.870412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.130:43869 (local address 127.21.12.190:36197)
0806 06:27:16.870477 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0806 06:27:16.870479 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:27:16.870708 (+ 229us) server_negotiation.cc:407] Connection header received
0806 06:27:16.871124 (+ 416us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:27:16.871125 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:27:16.871130 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:27:16.871176 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:27:16.872370 (+ 1194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:27:16.872372 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:27:16.872372 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 06:27:16.872376 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:27:16.872392 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 06:27:16.872396 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:27:16.872445 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0806 06:27:16.872527 (+ 82us) server_negotiation.cc:299] Negotiation successful
0806 06:27:16.872541 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260806 06:27:16.872762 3911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:27:16.870407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:36197 (local address 127.21.12.130:43869)
0806 06:27:16.870592 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0806 06:27:16.870625 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0806 06:27:16.871055 (+ 430us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:27:16.871232 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:27:16.871235 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:27:16.871243 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0806 06:27:16.872134 (+ 891us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 06:27:16.872135 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0806 06:27:16.872168 (+ 33us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 06:27:16.872204 (+ 36us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 06:27:16.872453 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 06:27:16.872458 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 06:27:16.872461 (+ 3us) client_negotiation.cc:769] Sending connection context
0806 06:27:16.872530 (+ 69us) client_negotiation.cc:240] Negotiation successful
0806 06:27:16.872579 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":84,"threads_started":1}
I20260806 06:27:16.873138 21554 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHhLhT/build/release/bin/kudu
/tmp/dist-test-taskFHhLhT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785997581049705-21554-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785997581049705-21554-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785997581049705-21554-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785997581049705-21554-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.12.131:0
--local_ip_for_outbound_sockets=127.21.12.131
--webserver_interface=127.21.12.131
--webserver_port=0
--tserver_master_addrs=127.21.12.190:36197
--builtin_ntp_servers=127.21.12.148:34657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260806 06:27:16.873175 3909 heartbeater.cc:347] Connected to a master server at 127.21.12.190:36197
I20260806 06:27:16.873255 3909 heartbeater.cc:464] Registering TS with master...
I20260806 06:27:16.873386 3909 heartbeater.cc:511] Master 127.21.12.190:36197 requested a full tablet report, sending...
I20260806 06:27:16.873725 3429 ts_manager.cc:194] Registered new tserver with Master: 0e1012eecb414738b0d96589e5b48054 (127.21.12.130:42575)
W20260806 06:27:16.953244 3912 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 06:27:16.953429 3912 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 06:27:16.953449 3912 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 06:27:16.954640 3912 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 06:27:16.954677 3912 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 06:27:16.954701 3912 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.12.131
I20260806 06:27:16.956009 3912 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.148:34657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785997581049705-21554-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785997581049705-21554-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.12.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785997581049705-21554-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.12.131
--webserver_port=0
--tserver_master_addrs=127.21.12.190:36197
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3912
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.12.131
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785997581049705-21554-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 05:11:02 UTC on ccycloud.quasar-hvzbbz.root.comops.site
build id 4569617
I20260806 06:27:16.956185 3912 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 06:27:16.956405 3912 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 06:27:16.961244 3922 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:27:16.961220 3924 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:27:16.961221 3921 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:27:16.961483 3912 server_base.cc:1034] running on GCE node
I20260806 06:27:16.961694 3912 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 06:27:16.961917 3912 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 06:27:16.963057 3912 hybrid_clock.cc:648] HybridClock initialized: now 1785997636963035 us; error 34 us; skew 500 ppm
I20260806 06:27:16.964222 3912 webserver.cc:460] Webserver started at http://127.21.12.131:43403/ using document root <none> and password file <none>
I20260806 06:27:16.964398 3912 fs_manager.cc:359] Metadata directory not provided
I20260806 06:27:16.964435 3912 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 06:27:16.964529 3912 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 06:27:16.965534 3912 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785997581049705-21554-0/minicluster-data/ts-2/data/instance:
uuid: "e9a55051923345e6a1f0482d0489978e"
format_stamp: "Formatted at 2026-08-06 06:27:16 on dist-test-slave-4tdj"
server_key: "6103ce9b1325d5b1186e2cffe099973a"
server_key_iv: "1ba881533c79463d09ce3bca88478ce2"
server_key_version: "encryptionkey@0"
I20260806 06:27:16.965837 3912 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785997581049705-21554-0/minicluster-data/ts-2/wal/instance:
uuid: "e9a55051923345e6a1f0482d0489978e"
format_stamp: "Formatted at 2026-08-06 06:27:16 on dist-test-slave-4tdj"
server_key: "6103ce9b1325d5b1186e2cffe099973a"
server_key_iv: "1ba881533c79463d09ce3bca88478ce2"
server_key_version: "encryptionkey@0"
I20260806 06:27:16.967037 3912 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260806 06:27:16.967635 3935 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 06:27:16.967794 3912 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260806 06:27:16.967883 3912 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785997581049705-21554-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785997581049705-21554-0/minicluster-data/ts-2/wal
uuid: "e9a55051923345e6a1f0482d0489978e"
format_stamp: "Formatted at 2026-08-06 06:27:16 on dist-test-slave-4tdj"
server_key: "6103ce9b1325d5b1186e2cffe099973a"
server_key_iv: "1ba881533c79463d09ce3bca88478ce2"
server_key_version: "encryptionkey@0"
I20260806 06:27:16.967938 3912 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785997581049705-21554-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785997581049705-21554-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785997581049705-21554-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 06:27:16.971101 3912 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 06:27:16.971302 3912 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 06:27:16.971390 3912 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 06:27:16.971539 3912 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 06:27:16.971767 3912 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 06:27:16.971796 3912 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 06:27:16.971813 3912 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 06:27:16.971830 3912 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 06:27:16.976989 3912 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.131:34957
I20260806 06:27:16.977043 4111 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.131:34957 every 8 connection(s)
I20260806 06:27:16.977301 3912 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785997581049705-21554-0/minicluster-data/ts-2/data/info.pb
I20260806 06:27:16.978065 21554 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHhLhT/build/release/bin/kudu as pid 3912
I20260806 06:27:16.978137 21554 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785997581049705-21554-0/minicluster-data/ts-2/wal/instance
I20260806 06:27:16.978349 21554 external_mini_cluster.cc:1428] Setting key 4b29e4b1390fff9b324406d5cab3bd10
I20260806 06:27:16.980050 3486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:27:16.977931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.131:44991 (local address 127.21.12.190:36197)
0806 06:27:16.977990 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0806 06:27:16.977993 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 06:27:16.978192 (+ 199us) server_negotiation.cc:407] Connection header received
0806 06:27:16.978763 (+ 571us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:27:16.978764 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:27:16.978771 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:27:16.978816 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:27:16.979874 (+ 1058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:27:16.979875 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:27:16.979875 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 06:27:16.979879 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:27:16.979893 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 06:27:16.979896 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:27:16.979937 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0806 06:27:16.979987 (+ 50us) server_negotiation.cc:299] Negotiation successful
0806 06:27:16.979999 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260806 06:27:16.980221 4114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:27:16.977903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:36197 (local address 127.21.12.131:44991)
0806 06:27:16.978077 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0806 06:27:16.978115 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0806 06:27:16.978679 (+ 564us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:27:16.978886 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:27:16.978890 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:27:16.978900 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0806 06:27:16.979746 (+ 846us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 06:27:16.979746 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0806 06:27:16.979773 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 06:27:16.979799 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 06:27:16.979938 (+ 139us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 06:27:16.979940 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 06:27:16.979942 (+ 2us) client_negotiation.cc:769] Sending connection context
0806 06:27:16.979986 (+ 44us) client_negotiation.cc:240] Negotiation successful
0806 06:27:16.980038 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":76,"threads_started":1}
I20260806 06:27:16.980624 4112 heartbeater.cc:347] Connected to a master server at 127.21.12.190:36197
I20260806 06:27:16.980708 4112 heartbeater.cc:464] Registering TS with master...
I20260806 06:27:16.980841 4112 heartbeater.cc:511] Master 127.21.12.190:36197 requested a full tablet report, sending...
I20260806 06:27:16.981153 3429 ts_manager.cc:194] Registered new tserver with Master: e9a55051923345e6a1f0482d0489978e (127.21.12.131:34957)
I20260806 06:27:16.989975 21554 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260806 06:27:16.991667 3486 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 06:27:16.991207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33370 (local address 127.21.12.190:36197)
0806 06:27:16.991269 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0806 06:27:16.991271 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:27:16.991362 (+ 91us) server_negotiation.cc:407] Connection header received
0806 06:27:16.991420 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:27:16.991421 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:27:16.991426 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:27:16.991469 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:27:16.991605 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33370: BlockingRecv error: recv got EOF from 127.0.0.1:33370 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
I20260806 06:27:16.991729 21554 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHhLhT/build/release/bin/kudu with pid 3514
I20260806 06:27:17.002929 21554 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHhLhT/build/release/bin/kudu with pid 3721
I20260806 06:27:17.012030 21554 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHhLhT/build/release/bin/kudu with pid 3912
I20260806 06:27:17.026870 21554 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHhLhT/build/release/bin/kudu with pid 3382
2026-08-06T06:27:17Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (544 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785997581049705-21554-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 06:27:17 dist-test-slave-4tdj krb5kdc[4126](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 06:27:17 dist-test-slave-4tdj krb5kdc[4126](info): set up 2 sockets
Aug 06 06:27:17 dist-test-slave-4tdj krb5kdc[4126](info): commencing operation
krb5kdc: starting...
W20260806 06:27:18.139885 21554 mini_kdc.cc:121] Time spent starting KDC: real 1.093s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785997581049705-21554-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785997581049705-21554-0/krb5kdc/test-user.keytab.
Aug 06 06:27:18 dist-test-slave-4tdj krb5kdc[4126](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997638, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T06:27:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T06:27:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.12.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.12.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.12.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.190.keytab.
Entry for principal kudu/127.21.12.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.190.keytab.
I20260806 06:27:18.256345 21554 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHhLhT/build/release/bin/kudu
/tmp/dist-test-taskFHhLhT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785997581049705-21554-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785997581049705-21554-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785997581049705-21554-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785997581049705-21554-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.12.190:40289
--webserver_interface=127.21.12.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.190.keytab
--principal=kudu/127.21.12.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.148:34069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.12.190:40289
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785997581049705-21554-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785997581049705-21554-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785997581049705-21554-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 06:27:18.335569 4150 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 06:27:18.335778 4150 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 06:27:18.335809 4150 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 06:27:18.336915 4150 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 06:27:18.336954 4150 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 06:27:18.336972 4150 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 06:27:18.336989 4150 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 06:27:18.337003 4150 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 06:27:18.338485 4150 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.148:34069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785997581049705-21554-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785997581049705-21554-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.12.190:40289
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.190.keytab
--principal=kudu/127.21.12.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.12.190:40289
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785997581049705-21554-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.12.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4150
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785997581049705-21554-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 05:11:02 UTC on ccycloud.quasar-hvzbbz.root.comops.site
build id 4569617
I20260806 06:27:18.338732 4150 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 06:27:18.338932 4150 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 06:27:18.343621 4160 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:27:18.343688 4158 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:27:18.343648 4157 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:27:18.343797 4150 server_base.cc:1034] running on GCE node
I20260806 06:27:18.344008 4150 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 06:27:18.344249 4150 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 06:27:18.345413 4150 hybrid_clock.cc:648] HybridClock initialized: now 1785997638345404 us; error 32 us; skew 500 ppm
Aug 06 06:27:18 dist-test-slave-4tdj krb5kdc[4126](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997638, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 06:27:18.347848 4150 init.cc:377] Logged in from keytab as kudu/127.21.12.190@KRBTEST.COM (short username kudu)
I20260806 06:27:18.348256 4150 webserver.cc:460] Webserver started at http://127.21.12.190:33107/ using document root <none> and password file <none>
I20260806 06:27:18.348464 4150 fs_manager.cc:359] Metadata directory not provided
I20260806 06:27:18.348507 4150 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 06:27:18.348649 4150 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 06:27:18.349663 4150 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785997581049705-21554-0/minicluster-data/master-0/data/instance:
uuid: "15963a3d7a204faaa545504b24a5ebfc"
format_stamp: "Formatted at 2026-08-06 06:27:18 on dist-test-slave-4tdj"
server_key: "894c15b117206dc186253510b8e11bf3"
server_key_iv: "1ac835b508389bb741d5cd4632dea8b6"
server_key_version: "encryptionkey@0"
I20260806 06:27:18.349987 4150 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785997581049705-21554-0/minicluster-data/master-0/wal/instance:
uuid: "15963a3d7a204faaa545504b24a5ebfc"
format_stamp: "Formatted at 2026-08-06 06:27:18 on dist-test-slave-4tdj"
server_key: "894c15b117206dc186253510b8e11bf3"
server_key_iv: "1ac835b508389bb741d5cd4632dea8b6"
server_key_version: "encryptionkey@0"
I20260806 06:27:18.351257 4150 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260806 06:27:18.351918 4172 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 06:27:18.352051 4150 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260806 06:27:18.352134 4150 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785997581049705-21554-0/minicluster-data/master-0/data,/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785997581049705-21554-0/minicluster-data/master-0/wal
uuid: "15963a3d7a204faaa545504b24a5ebfc"
format_stamp: "Formatted at 2026-08-06 06:27:18 on dist-test-slave-4tdj"
server_key: "894c15b117206dc186253510b8e11bf3"
server_key_iv: "1ac835b508389bb741d5cd4632dea8b6"
server_key_version: "encryptionkey@0"
I20260806 06:27:18.352200 4150 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785997581049705-21554-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785997581049705-21554-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785997581049705-21554-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 06:27:18.389621 4150 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 06:27:18.391525 4150 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 06:27:18.391665 4150 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 06:27:18.395381 4150 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.190:40289
I20260806 06:27:18.395432 4250 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.190:40289 every 8 connection(s)
I20260806 06:27:18.395750 4150 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785997581049705-21554-0/minicluster-data/master-0/data/info.pb
I20260806 06:27:18.396696 4253 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 06:27:18.398919 4253 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 15963a3d7a204faaa545504b24a5ebfc: Bootstrap starting.
I20260806 06:27:18.399483 4253 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 15963a3d7a204faaa545504b24a5ebfc: Neither blocks nor log segments found. Creating new log.
I20260806 06:27:18.399748 4253 log.cc:824] T 00000000000000000000000000000000 P 15963a3d7a204faaa545504b24a5ebfc: Log is configured to *not* fsync() on all Append() calls
I20260806 06:27:18.400390 4253 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 15963a3d7a204faaa545504b24a5ebfc: No bootstrap required, opened a new log
I20260806 06:27:18.401314 21554 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHhLhT/build/release/bin/kudu as pid 4150
I20260806 06:27:18.401410 21554 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785997581049705-21554-0/minicluster-data/master-0/wal/instance
I20260806 06:27:18.401611 21554 external_mini_cluster.cc:1428] Setting key a3663f9b3d0a47ebac0f1f3a92cb31d9
I20260806 06:27:18.401997 4253 raft_consensus.cc:348] T 00000000000000000000000000000000 P 15963a3d7a204faaa545504b24a5ebfc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15963a3d7a204faaa545504b24a5ebfc" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 40289 } }
I20260806 06:27:18.402185 4253 raft_consensus.cc:374] T 00000000000000000000000000000000 P 15963a3d7a204faaa545504b24a5ebfc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 06:27:18.402235 4253 raft_consensus.cc:729] T 00000000000000000000000000000000 P 15963a3d7a204faaa545504b24a5ebfc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15963a3d7a204faaa545504b24a5ebfc, State: Initialized, Role: FOLLOWER
I20260806 06:27:18.402392 4253 consensus_queue.cc:260] T 00000000000000000000000000000000 P 15963a3d7a204faaa545504b24a5ebfc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15963a3d7a204faaa545504b24a5ebfc" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 40289 } }
I20260806 06:27:18.402464 4253 raft_consensus.cc:388] T 00000000000000000000000000000000 P 15963a3d7a204faaa545504b24a5ebfc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 06:27:18.402485 4253 raft_consensus.cc:482] T 00000000000000000000000000000000 P 15963a3d7a204faaa545504b24a5ebfc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 06:27:18.402532 4253 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 15963a3d7a204faaa545504b24a5ebfc [term 0 FOLLOWER]: Advancing to term 1
I20260806 06:27:18.403553 4253 raft_consensus.cc:504] T 00000000000000000000000000000000 P 15963a3d7a204faaa545504b24a5ebfc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15963a3d7a204faaa545504b24a5ebfc" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 40289 } }
I20260806 06:27:18.403644 4253 leader_election.cc:302] T 00000000000000000000000000000000 P 15963a3d7a204faaa545504b24a5ebfc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 15963a3d7a204faaa545504b24a5ebfc; no voters:
I20260806 06:27:18.403795 4253 leader_election.cc:288] T 00000000000000000000000000000000 P 15963a3d7a204faaa545504b24a5ebfc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 06:27:18.403848 4258 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 15963a3d7a204faaa545504b24a5ebfc [term 1 FOLLOWER]: Leader election won for term 1
I20260806 06:27:18.403997 4258 raft_consensus.cc:686] T 00000000000000000000000000000000 P 15963a3d7a204faaa545504b24a5ebfc [term 1 LEADER]: Becoming Leader. State: Replica: 15963a3d7a204faaa545504b24a5ebfc, State: Running, Role: LEADER
I20260806 06:27:18.404048 4253 sys_catalog.cc:564] T 00000000000000000000000000000000 P 15963a3d7a204faaa545504b24a5ebfc [sys.catalog]: configured and running, proceeding with master startup.
I20260806 06:27:18.404145 4258 consensus_queue.cc:237] T 00000000000000000000000000000000 P 15963a3d7a204faaa545504b24a5ebfc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15963a3d7a204faaa545504b24a5ebfc" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 40289 } }
I20260806 06:27:18.404462 4258 sys_catalog.cc:455] T 00000000000000000000000000000000 P 15963a3d7a204faaa545504b24a5ebfc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 15963a3d7a204faaa545504b24a5ebfc. Latest consensus state: current_term: 1 leader_uuid: "15963a3d7a204faaa545504b24a5ebfc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15963a3d7a204faaa545504b24a5ebfc" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 40289 } } }
I20260806 06:27:18.404474 4260 sys_catalog.cc:455] T 00000000000000000000000000000000 P 15963a3d7a204faaa545504b24a5ebfc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "15963a3d7a204faaa545504b24a5ebfc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15963a3d7a204faaa545504b24a5ebfc" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 40289 } } }
I20260806 06:27:18.404641 4258 sys_catalog.cc:458] T 00000000000000000000000000000000 P 15963a3d7a204faaa545504b24a5ebfc [sys.catalog]: This master's current role is: LEADER
I20260806 06:27:18.404878 4260 sys_catalog.cc:458] T 00000000000000000000000000000000 P 15963a3d7a204faaa545504b24a5ebfc [sys.catalog]: This master's current role is: LEADER
W20260806 06:27:18.405943 4287 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 15963a3d7a204faaa545504b24a5ebfc: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 06:27:18.406047 4287 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 06:27:18.406764 4276 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 06:27:18.406965 4276 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Aug 06 06:27:18 dist-test-slave-4tdj krb5kdc[4126](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997638, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.12.190@KRBTEST.COM
I20260806 06:27:18.408459 4276 catalog_manager.cc:1347] Generated new cluster ID: dabc4a737fba44bb9e01b99edd70b869
I20260806 06:27:18.408504 4276 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 06:27:18.412997 4257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:27:18.402123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60692 (local address 127.21.12.190:40289)
0806 06:27:18.402341 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0806 06:27:18.402351 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 06:27:18.402386 (+ 35us) server_negotiation.cc:407] Connection header received
0806 06:27:18.402504 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:27:18.402512 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:27:18.402568 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:27:18.402691 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:27:18.403435 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:27:18.404662 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:27:18.405739 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:27:18.406277 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:27:18.406398 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:27:18.409010 (+ 2612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:27:18.409015 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:27:18.409015 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 06:27:18.409026 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:27:18.411419 (+ 2393us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:27:18.411814 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:27:18.411817 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:27:18.411817 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:27:18.411936 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:27:18.412132 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:27:18.412133 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:27:18.412133 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:27:18.412415 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:27:18.412567 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0806 06:27:18.412664 (+ 97us) server_negotiation.cc:299] Negotiation successful
0806 06:27:18.412789 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":77,"threads_started":1}
I20260806 06:27:18.424325 4276 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 06:27:18.424889 4276 catalog_manager.cc:1504] Loading token signing keys...
I20260806 06:27:18.430576 4276 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 15963a3d7a204faaa545504b24a5ebfc: Generated new TSK 0
I20260806 06:27:18.430750 4276 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.12.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.12.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.12.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.129.keytab.
Entry for principal kudu/127.21.12.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.129.keytab.
I20260806 06:27:18.456619 21554 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHhLhT/build/release/bin/kudu
/tmp/dist-test-taskFHhLhT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785997581049705-21554-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785997581049705-21554-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785997581049705-21554-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785997581049705-21554-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.12.129:0
--local_ip_for_outbound_sockets=127.21.12.129
--webserver_interface=127.21.12.129
--webserver_port=0
--tserver_master_addrs=127.21.12.190:40289
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.129.keytab
--principal=kudu/127.21.12.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.148:34069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785997581049705-21554-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785997581049705-21554-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785997581049705-21554-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 06:27:18.536612 4293 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 06:27:18.536866 4293 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 06:27:18.536899 4293 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 06:27:18.538398 4293 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 06:27:18.538440 4293 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 06:27:18.538467 4293 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.12.129
I20260806 06:27:18.540146 4293 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.148:34069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785997581049705-21554-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785997581049705-21554-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.129.keytab
--principal=kudu/127.21.12.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.12.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785997581049705-21554-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.12.129
--webserver_port=0
--tserver_master_addrs=127.21.12.190:40289
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4293
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.12.129
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785997581049705-21554-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 05:11:02 UTC on ccycloud.quasar-hvzbbz.root.comops.site
build id 4569617
I20260806 06:27:18.540377 4293 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 06:27:18.540589 4293 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 06:27:18.545487 4301 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:27:18.545555 4293 server_base.cc:1034] running on GCE node
W20260806 06:27:18.545507 4306 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:27:18.545471 4302 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:27:18.545925 4293 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 06:27:18.546121 4293 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 06:27:18.547253 4293 hybrid_clock.cc:648] HybridClock initialized: now 1785997638547236 us; error 36 us; skew 500 ppm
Aug 06 06:27:18 dist-test-slave-4tdj krb5kdc[4126](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997638, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 06:27:18.549609 4293 init.cc:377] Logged in from keytab as kudu/127.21.12.129@KRBTEST.COM (short username kudu)
I20260806 06:27:18.550017 4293 webserver.cc:460] Webserver started at http://127.21.12.129:45593/ using document root <none> and password file <none>
I20260806 06:27:18.550207 4293 fs_manager.cc:359] Metadata directory not provided
I20260806 06:27:18.550256 4293 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 06:27:18.550400 4293 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 06:27:18.551393 4293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785997581049705-21554-0/minicluster-data/ts-0/data/instance:
uuid: "7c09c22b0380440ba8ca1759a24c59f2"
format_stamp: "Formatted at 2026-08-06 06:27:18 on dist-test-slave-4tdj"
server_key: "54924093fb42fba3405cdbcc803ce885"
server_key_iv: "7c46320d86ef26eaade6736c8e5ffb54"
server_key_version: "encryptionkey@0"
I20260806 06:27:18.551720 4293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785997581049705-21554-0/minicluster-data/ts-0/wal/instance:
uuid: "7c09c22b0380440ba8ca1759a24c59f2"
format_stamp: "Formatted at 2026-08-06 06:27:18 on dist-test-slave-4tdj"
server_key: "54924093fb42fba3405cdbcc803ce885"
server_key_iv: "7c46320d86ef26eaade6736c8e5ffb54"
server_key_version: "encryptionkey@0"
I20260806 06:27:18.552915 4293 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260806 06:27:18.553607 4316 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 06:27:18.553755 4293 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260806 06:27:18.553838 4293 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785997581049705-21554-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785997581049705-21554-0/minicluster-data/ts-0/wal
uuid: "7c09c22b0380440ba8ca1759a24c59f2"
format_stamp: "Formatted at 2026-08-06 06:27:18 on dist-test-slave-4tdj"
server_key: "54924093fb42fba3405cdbcc803ce885"
server_key_iv: "7c46320d86ef26eaade6736c8e5ffb54"
server_key_version: "encryptionkey@0"
I20260806 06:27:18.553897 4293 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785997581049705-21554-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785997581049705-21554-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785997581049705-21554-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 06:27:18.568524 4293 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 06:27:18.570196 4293 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 06:27:18.570351 4293 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 06:27:18.570559 4293 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 06:27:18.570861 4293 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 06:27:18.570895 4293 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 06:27:18.570924 4293 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 06:27:18.570945 4293 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 06:27:18.576141 4293 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.129:46441
I20260806 06:27:18.576205 4479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.129:46441 every 8 connection(s)
I20260806 06:27:18.576478 4293 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785997581049705-21554-0/minicluster-data/ts-0/data/info.pb
I20260806 06:27:18.581719 21554 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHhLhT/build/release/bin/kudu as pid 4293
I20260806 06:27:18.581796 21554 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785997581049705-21554-0/minicluster-data/ts-0/wal/instance
I20260806 06:27:18.581969 21554 external_mini_cluster.cc:1428] Setting key 7eb86ab9d168d1896a76f1e6aa16c2af
Aug 06 06:27:18 dist-test-slave-4tdj krb5kdc[4126](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997638, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.129@KRBTEST.COM for kudu/127.21.12.190@KRBTEST.COM
I20260806 06:27:18.585071 4257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:27:18.577142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.129:50163 (local address 127.21.12.190:40289)
0806 06:27:18.577201 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0806 06:27:18.577203 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:27:18.577425 (+ 222us) server_negotiation.cc:407] Connection header received
0806 06:27:18.577865 (+ 440us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:27:18.577866 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:27:18.577882 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:27:18.577938 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:27:18.578845 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:27:18.579341 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:27:18.580260 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:27:18.580468 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:27:18.580529 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:27:18.582489 (+ 1960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:27:18.582490 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:27:18.582491 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 06:27:18.582498 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:27:18.583810 (+ 1312us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:27:18.584244 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:27:18.584245 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:27:18.584245 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:27:18.584311 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:27:18.584501 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:27:18.584502 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:27:18.584502 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:27:18.584635 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:27:18.584695 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0806 06:27:18.584901 (+ 206us) server_negotiation.cc:299] Negotiation successful
0806 06:27:18.584977 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260806 06:27:18.585134 4483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:27:18.577119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:40289 (local address 127.21.12.129:50163)
0806 06:27:18.577279 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0806 06:27:18.577319 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0806 06:27:18.577786 (+ 467us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:27:18.577996 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:27:18.578000 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:27:18.578582 (+ 582us) client_negotiation.cc:189] Negotiated authn=SASL
0806 06:27:18.578776 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:27:18.578780 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:27:18.579404 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:27:18.579405 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:27:18.580166 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:27:18.580168 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:27:18.580534 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:27:18.580535 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:27:18.580625 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:27:18.581029 (+ 404us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 06:27:18.581037 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0806 06:27:18.582398 (+ 1361us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 06:27:18.583891 (+ 1493us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:27:18.583899 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:27:18.583901 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:27:18.584180 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:27:18.584368 (+ 188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:27:18.584369 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:27:18.584369 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:27:18.584456 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:27:18.584691 (+ 235us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 06:27:18.584694 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 06:27:18.584794 (+ 100us) client_negotiation.cc:769] Sending connection context
0806 06:27:18.584868 (+ 74us) client_negotiation.cc:240] Negotiation successful
0806 06:27:18.584976 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":79,"threads_started":1}
I20260806 06:27:18.585851 4480 heartbeater.cc:347] Connected to a master server at 127.21.12.190:40289
I20260806 06:27:18.585937 4480 heartbeater.cc:464] Registering TS with master...
I20260806 06:27:18.586102 4480 heartbeater.cc:511] Master 127.21.12.190:40289 requested a full tablet report, sending...
I20260806 06:27:18.586660 4193 ts_manager.cc:194] Registered new tserver with Master: 7c09c22b0380440ba8ca1759a24c59f2 (127.21.12.129:46441)
I20260806 06:27:18.587560 4193 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.12.129@KRBTEST.COM'} at 127.21.12.129:50163
WARNING: no policy specified for kudu/127.21.12.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.12.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.12.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.130.keytab.
Entry for principal kudu/127.21.12.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.130.keytab.
I20260806 06:27:18.606914 21554 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHhLhT/build/release/bin/kudu
/tmp/dist-test-taskFHhLhT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785997581049705-21554-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785997581049705-21554-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785997581049705-21554-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785997581049705-21554-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.12.130:0
--local_ip_for_outbound_sockets=127.21.12.130
--webserver_interface=127.21.12.130
--webserver_port=0
--tserver_master_addrs=127.21.12.190:40289
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.130.keytab
--principal=kudu/127.21.12.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.148:34069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785997581049705-21554-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785997581049705-21554-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785997581049705-21554-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 06:27:18.687255 4486 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 06:27:18.687460 4486 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 06:27:18.687482 4486 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 06:27:18.688623 4486 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 06:27:18.688668 4486 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 06:27:18.688694 4486 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.12.130
I20260806 06:27:18.690179 4486 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.148:34069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785997581049705-21554-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785997581049705-21554-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.130.keytab
--principal=kudu/127.21.12.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.12.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785997581049705-21554-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.12.130
--webserver_port=0
--tserver_master_addrs=127.21.12.190:40289
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4486
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.12.130
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785997581049705-21554-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 05:11:02 UTC on ccycloud.quasar-hvzbbz.root.comops.site
build id 4569617
I20260806 06:27:18.690426 4486 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 06:27:18.690641 4486 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 06:27:18.695677 4500 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:27:18.695713 4496 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:27:18.695721 4497 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:27:18.695771 4486 server_base.cc:1034] running on GCE node
I20260806 06:27:18.696048 4486 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 06:27:18.696272 4486 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 06:27:18.697432 4486 hybrid_clock.cc:648] HybridClock initialized: now 1785997638697419 us; error 35 us; skew 500 ppm
Aug 06 06:27:18 dist-test-slave-4tdj krb5kdc[4126](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997638, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 06:27:18.699852 4486 init.cc:377] Logged in from keytab as kudu/127.21.12.130@KRBTEST.COM (short username kudu)
I20260806 06:27:18.700269 4486 webserver.cc:460] Webserver started at http://127.21.12.130:42941/ using document root <none> and password file <none>
I20260806 06:27:18.700469 4486 fs_manager.cc:359] Metadata directory not provided
I20260806 06:27:18.700507 4486 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 06:27:18.700600 4486 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 06:27:18.701570 4486 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785997581049705-21554-0/minicluster-data/ts-1/data/instance:
uuid: "3d853f8c7420488b895f2ec644e36ee6"
format_stamp: "Formatted at 2026-08-06 06:27:18 on dist-test-slave-4tdj"
server_key: "f0372b6b37aa307afba1b6856c7df882"
server_key_iv: "e47e49d42e0360178f6118cd15bfcf1a"
server_key_version: "encryptionkey@0"
I20260806 06:27:18.701853 4486 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785997581049705-21554-0/minicluster-data/ts-1/wal/instance:
uuid: "3d853f8c7420488b895f2ec644e36ee6"
format_stamp: "Formatted at 2026-08-06 06:27:18 on dist-test-slave-4tdj"
server_key: "f0372b6b37aa307afba1b6856c7df882"
server_key_iv: "e47e49d42e0360178f6118cd15bfcf1a"
server_key_version: "encryptionkey@0"
I20260806 06:27:18.703068 4486 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260806 06:27:18.703694 4510 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 06:27:18.703855 4486 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260806 06:27:18.703927 4486 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785997581049705-21554-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785997581049705-21554-0/minicluster-data/ts-1/wal
uuid: "3d853f8c7420488b895f2ec644e36ee6"
format_stamp: "Formatted at 2026-08-06 06:27:18 on dist-test-slave-4tdj"
server_key: "f0372b6b37aa307afba1b6856c7df882"
server_key_iv: "e47e49d42e0360178f6118cd15bfcf1a"
server_key_version: "encryptionkey@0"
I20260806 06:27:18.703979 4486 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785997581049705-21554-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785997581049705-21554-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785997581049705-21554-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 06:27:18.715619 4486 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 06:27:18.717581 4486 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 06:27:18.717691 4486 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 06:27:18.717873 4486 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 06:27:18.718111 4486 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 06:27:18.718139 4486 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 06:27:18.718158 4486 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 06:27:18.718182 4486 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 06:27:18.723832 4486 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.130:42265
I20260806 06:27:18.723945 4680 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.130:42265 every 8 connection(s)
I20260806 06:27:18.724181 4486 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785997581049705-21554-0/minicluster-data/ts-1/data/info.pb
Aug 06 06:27:18 dist-test-slave-4tdj krb5kdc[4126](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997638, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.130@KRBTEST.COM for kudu/127.21.12.190@KRBTEST.COM
I20260806 06:27:18.731580 21554 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHhLhT/build/release/bin/kudu as pid 4486
I20260806 06:27:18.731668 21554 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785997581049705-21554-0/minicluster-data/ts-1/wal/instance
I20260806 06:27:18.731863 21554 external_mini_cluster.cc:1428] Setting key da1d01411d801a50d18b9caf4657d2a8
I20260806 06:27:18.732685 4257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:27:18.724871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.130:34531 (local address 127.21.12.190:40289)
0806 06:27:18.724953 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0806 06:27:18.724955 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:27:18.725120 (+ 165us) server_negotiation.cc:407] Connection header received
0806 06:27:18.725584 (+ 464us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:27:18.725585 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:27:18.725595 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:27:18.725641 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:27:18.726570 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:27:18.727056 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:27:18.727976 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:27:18.728171 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:27:18.728234 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:27:18.730165 (+ 1931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:27:18.730167 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:27:18.730167 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 06:27:18.730173 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:27:18.731420 (+ 1247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:27:18.731807 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:27:18.731808 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:27:18.731808 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:27:18.731860 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:27:18.732084 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:27:18.732085 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:27:18.732086 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:27:18.732202 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:27:18.732263 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0806 06:27:18.732528 (+ 265us) server_negotiation.cc:299] Negotiation successful
0806 06:27:18.732615 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260806 06:27:18.732882 4688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:27:18.724844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:40289 (local address 127.21.12.130:34531)
0806 06:27:18.725007 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0806 06:27:18.725041 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0806 06:27:18.725507 (+ 466us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:27:18.725709 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:27:18.725713 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:27:18.726245 (+ 532us) client_negotiation.cc:189] Negotiated authn=SASL
0806 06:27:18.726489 (+ 244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:27:18.726495 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:27:18.727122 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:27:18.727123 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:27:18.727885 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:27:18.727887 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:27:18.728239 (+ 352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:27:18.728240 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:27:18.728331 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:27:18.728768 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 06:27:18.728782 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0806 06:27:18.730084 (+ 1302us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 06:27:18.731498 (+ 1414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:27:18.731506 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:27:18.731508 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:27:18.731746 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:27:18.731913 (+ 167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:27:18.731914 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:27:18.731914 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:27:18.732028 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:27:18.732274 (+ 246us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 06:27:18.732280 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 06:27:18.732395 (+ 115us) client_negotiation.cc:769] Sending connection context
0806 06:27:18.732501 (+ 106us) client_negotiation.cc:240] Negotiation successful
0806 06:27:18.732652 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":92,"threads_started":1}
I20260806 06:27:18.733271 4684 heartbeater.cc:347] Connected to a master server at 127.21.12.190:40289
I20260806 06:27:18.733356 4684 heartbeater.cc:464] Registering TS with master...
I20260806 06:27:18.733528 4684 heartbeater.cc:511] Master 127.21.12.190:40289 requested a full tablet report, sending...
I20260806 06:27:18.733891 4193 ts_manager.cc:194] Registered new tserver with Master: 3d853f8c7420488b895f2ec644e36ee6 (127.21.12.130:42265)
I20260806 06:27:18.734328 4193 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.12.130@KRBTEST.COM'} at 127.21.12.130:34531
WARNING: no policy specified for kudu/127.21.12.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.12.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.12.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.131.keytab.
Entry for principal kudu/127.21.12.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.131.keytab.
I20260806 06:27:18.756377 21554 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHhLhT/build/release/bin/kudu
/tmp/dist-test-taskFHhLhT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785997581049705-21554-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785997581049705-21554-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785997581049705-21554-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785997581049705-21554-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.12.131:0
--local_ip_for_outbound_sockets=127.21.12.131
--webserver_interface=127.21.12.131
--webserver_port=0
--tserver_master_addrs=127.21.12.190:40289
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.131.keytab
--principal=kudu/127.21.12.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.148:34069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785997581049705-21554-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785997581049705-21554-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785997581049705-21554-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 06:27:18.836839 4692 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 06:27:18.837037 4692 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 06:27:18.837059 4692 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 06:27:18.838114 4692 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 06:27:18.838147 4692 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 06:27:18.838171 4692 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.12.131
I20260806 06:27:18.839586 4692 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.148:34069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785997581049705-21554-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785997581049705-21554-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.131.keytab
--principal=kudu/127.21.12.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.12.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785997581049705-21554-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.12.131
--webserver_port=0
--tserver_master_addrs=127.21.12.190:40289
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4692
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.12.131
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785997581049705-21554-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 05:11:02 UTC on ccycloud.quasar-hvzbbz.root.comops.site
build id 4569617
I20260806 06:27:18.839774 4692 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 06:27:18.839943 4692 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 06:27:18.844615 4704 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:27:18.844699 4692 server_base.cc:1034] running on GCE node
W20260806 06:27:18.844599 4701 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:27:18.844587 4706 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:27:18.845039 4692 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 06:27:18.845293 4692 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 06:27:18.846428 4692 hybrid_clock.cc:648] HybridClock initialized: now 1785997638846418 us; error 29 us; skew 500 ppm
Aug 06 06:27:18 dist-test-slave-4tdj krb5kdc[4126](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997638, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 06:27:18.848659 4692 init.cc:377] Logged in from keytab as kudu/127.21.12.131@KRBTEST.COM (short username kudu)
I20260806 06:27:18.849057 4692 webserver.cc:460] Webserver started at http://127.21.12.131:33583/ using document root <none> and password file <none>
I20260806 06:27:18.849269 4692 fs_manager.cc:359] Metadata directory not provided
I20260806 06:27:18.849318 4692 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 06:27:18.849436 4692 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 06:27:18.850472 4692 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785997581049705-21554-0/minicluster-data/ts-2/data/instance:
uuid: "819c874b0d9e4d13a3c8fb593340484e"
format_stamp: "Formatted at 2026-08-06 06:27:18 on dist-test-slave-4tdj"
server_key: "e306ef8dd092d27ac5545a222aeca801"
server_key_iv: "92eaa5cc80563e2359bbfc0a065e006c"
server_key_version: "encryptionkey@0"
I20260806 06:27:18.850816 4692 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785997581049705-21554-0/minicluster-data/ts-2/wal/instance:
uuid: "819c874b0d9e4d13a3c8fb593340484e"
format_stamp: "Formatted at 2026-08-06 06:27:18 on dist-test-slave-4tdj"
server_key: "e306ef8dd092d27ac5545a222aeca801"
server_key_iv: "92eaa5cc80563e2359bbfc0a065e006c"
server_key_version: "encryptionkey@0"
I20260806 06:27:18.852020 4692 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260806 06:27:18.852663 4723 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 06:27:18.852842 4692 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260806 06:27:18.852927 4692 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785997581049705-21554-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785997581049705-21554-0/minicluster-data/ts-2/wal
uuid: "819c874b0d9e4d13a3c8fb593340484e"
format_stamp: "Formatted at 2026-08-06 06:27:18 on dist-test-slave-4tdj"
server_key: "e306ef8dd092d27ac5545a222aeca801"
server_key_iv: "92eaa5cc80563e2359bbfc0a065e006c"
server_key_version: "encryptionkey@0"
I20260806 06:27:18.852993 4692 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785997581049705-21554-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785997581049705-21554-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785997581049705-21554-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 06:27:18.864501 4692 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 06:27:18.866215 4692 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 06:27:18.866421 4692 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 06:27:18.866639 4692 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 06:27:18.866950 4692 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 06:27:18.866983 4692 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 06:27:18.867012 4692 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 06:27:18.867029 4692 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 06:27:18.872571 4692 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.131:43901
I20260806 06:27:18.872635 4881 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.131:43901 every 8 connection(s)
I20260806 06:27:18.872921 4692 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785997581049705-21554-0/minicluster-data/ts-2/data/info.pb
Aug 06 06:27:18 dist-test-slave-4tdj krb5kdc[4126](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997638, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.131@KRBTEST.COM for kudu/127.21.12.190@KRBTEST.COM
I20260806 06:27:18.881319 21554 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHhLhT/build/release/bin/kudu as pid 4692
I20260806 06:27:18.881420 21554 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785997581049705-21554-0/minicluster-data/ts-2/wal/instance
I20260806 06:27:18.881596 21554 external_mini_cluster.cc:1428] Setting key c92cc5a7fab8f850ef7e700800c6822b
I20260806 06:27:18.881803 4257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:27:18.873627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.131:34895 (local address 127.21.12.190:40289)
0806 06:27:18.873697 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0806 06:27:18.873699 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:27:18.873933 (+ 234us) server_negotiation.cc:407] Connection header received
0806 06:27:18.874529 (+ 596us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:27:18.874531 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:27:18.874544 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:27:18.874596 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:27:18.875542 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:27:18.876032 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:27:18.876992 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:27:18.877189 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:27:18.877243 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:27:18.879308 (+ 2065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:27:18.879309 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:27:18.879310 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 06:27:18.879317 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:27:18.880504 (+ 1187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:27:18.880877 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:27:18.880879 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:27:18.880879 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:27:18.880937 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:27:18.881171 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:27:18.881172 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:27:18.881173 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:27:18.881285 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:27:18.881340 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0806 06:27:18.881633 (+ 293us) server_negotiation.cc:299] Negotiation successful
0806 06:27:18.881713 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260806 06:27:18.881964 4888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:27:18.873585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:40289 (local address 127.21.12.131:34895)
0806 06:27:18.873779 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0806 06:27:18.873831 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0806 06:27:18.874422 (+ 591us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:27:18.874667 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:27:18.874671 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:27:18.875250 (+ 579us) client_negotiation.cc:189] Negotiated authn=SASL
0806 06:27:18.875438 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:27:18.875443 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:27:18.876113 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:27:18.876114 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:27:18.876906 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:27:18.876908 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:27:18.877257 (+ 349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:27:18.877258 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:27:18.877383 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:27:18.877860 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 06:27:18.877868 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0806 06:27:18.879218 (+ 1350us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 06:27:18.880592 (+ 1374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:27:18.880598 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:27:18.880601 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:27:18.880809 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:27:18.881000 (+ 191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:27:18.881001 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:27:18.881001 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:27:18.881107 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:27:18.881341 (+ 234us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 06:27:18.881348 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 06:27:18.881498 (+ 150us) client_negotiation.cc:769] Sending connection context
0806 06:27:18.881601 (+ 103us) client_negotiation.cc:240] Negotiation successful
0806 06:27:18.881756 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":78,"threads_started":1}
I20260806 06:27:18.882433 4886 heartbeater.cc:347] Connected to a master server at 127.21.12.190:40289
I20260806 06:27:18.882547 4886 heartbeater.cc:464] Registering TS with master...
I20260806 06:27:18.882725 4886 heartbeater.cc:511] Master 127.21.12.190:40289 requested a full tablet report, sending...
I20260806 06:27:18.883062 4193 ts_manager.cc:194] Registered new tserver with Master: 819c874b0d9e4d13a3c8fb593340484e (127.21.12.131:43901)
I20260806 06:27:18.883406 4193 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.12.131@KRBTEST.COM'} at 127.21.12.131:34895
I20260806 06:27:18.893486 21554 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 06:27:18.900089 4257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:27:18.894784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60694 (local address 127.21.12.190:40289)
0806 06:27:18.894834 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0806 06:27:18.894836 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:27:18.894938 (+ 102us) server_negotiation.cc:407] Connection header received
0806 06:27:18.894988 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:27:18.894988 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:27:18.894997 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:27:18.895051 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:27:18.895555 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:27:18.896016 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:27:18.896663 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:27:18.896835 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:27:18.896879 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:27:18.897901 (+ 1022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:27:18.897902 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:27:18.897902 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 06:27:18.897907 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:27:18.899239 (+ 1332us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:27:18.899496 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:27:18.899497 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:27:18.899498 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:27:18.899547 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:27:18.899665 (+ 118us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:27:18.899666 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:27:18.899666 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:27:18.899804 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:27:18.899869 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0806 06:27:18.899946 (+ 77us) server_negotiation.cc:299] Negotiation successful
0806 06:27:18.900002 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260806 06:27:18.901464 4193 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60694:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 06:27:18.903021 4193 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 06:27:18.909567 4896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:27:18.905187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.131:43901 (local address 127.0.0.1:55092)
0806 06:27:18.905624 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0806 06:27:18.905668 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0806 06:27:18.905735 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:27:18.906391 (+ 656us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:27:18.906392 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:27:18.906394 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 06:27:18.906430 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:27:18.906431 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:27:18.907709 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:27:18.907710 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:27:18.908587 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:27:18.908591 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:27:18.909288 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:27:18.909289 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:27:18.909364 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:27:18.909378 (+ 14us) client_negotiation.cc:769] Sending connection context
0806 06:27:18.909441 (+ 63us) client_negotiation.cc:240] Negotiation successful
0806 06:27:18.909491 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":85,"threads_started":1}
I20260806 06:27:18.909638 4899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:27:18.905261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.129:46441 (local address 127.0.0.1:38628)
0806 06:27:18.905875 (+ 614us) negotiation.cc:107] Waiting for socket to connect
0806 06:27:18.905885 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0806 06:27:18.905935 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:27:18.906204 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:27:18.906207 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:27:18.906209 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 06:27:18.906276 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:27:18.906278 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:27:18.907303 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:27:18.907304 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:27:18.908526 (+ 1222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:27:18.908529 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:27:18.909412 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:27:18.909413 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:27:18.909468 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:27:18.909471 (+ 3us) client_negotiation.cc:769] Sending connection context
0806 06:27:18.909517 (+ 46us) client_negotiation.cc:240] Negotiation successful
0806 06:27:18.909549 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":596,"thread_start_us":57,"threads_started":1}
I20260806 06:27:18.909823 4898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:27:18.905306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55092 (local address 127.21.12.131:43901)
0806 06:27:18.906163 (+ 857us) server_negotiation.cc:206] Beginning negotiation
0806 06:27:18.906167 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 06:27:18.906175 (+ 8us) server_negotiation.cc:407] Connection header received
0806 06:27:18.906208 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:27:18.906210 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:27:18.906224 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:27:18.906315 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 06:27:18.906497 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:27:18.907589 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:27:18.908687 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:27:18.909215 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:27:18.909278 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:27:18.909338 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0806 06:27:18.909700 (+ 362us) server_negotiation.cc:299] Negotiation successful
0806 06:27:18.909746 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":821,"thread_start_us":104,"threads_started":1}
I20260806 06:27:18.909871 4900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:27:18.905433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38628 (local address 127.21.12.129:46441)
0806 06:27:18.905599 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0806 06:27:18.905603 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 06:27:18.905934 (+ 331us) server_negotiation.cc:407] Connection header received
0806 06:27:18.906002 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:27:18.906004 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:27:18.906024 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:27:18.906096 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 06:27:18.906378 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:27:18.907216 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:27:18.908616 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:27:18.909322 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:27:18.909412 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:27:18.909477 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0806 06:27:18.909532 (+ 55us) server_negotiation.cc:299] Negotiation successful
0806 06:27:18.909577 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":77,"threads_started":1}
I20260806 06:27:18.910212 4897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:27:18.905422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.130:42265 (local address 127.0.0.1:55920)
0806 06:27:18.906002 (+ 580us) negotiation.cc:107] Waiting for socket to connect
0806 06:27:18.906010 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0806 06:27:18.906052 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:27:18.906278 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:27:18.906279 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:27:18.906282 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 06:27:18.906360 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:27:18.906363 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:27:18.907686 (+ 1323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:27:18.907687 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:27:18.908751 (+ 1064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:27:18.908754 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:27:18.909754 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:27:18.909755 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:27:18.909954 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:27:18.909970 (+ 16us) client_negotiation.cc:769] Sending connection context
0806 06:27:18.910097 (+ 127us) client_negotiation.cc:240] Negotiation successful
0806 06:27:18.910142 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":572,"mutex_wait_us":10,"thread_start_us":64,"threads_started":1}
I20260806 06:27:18.910493 4794 tablet_service.cc:1467] Processing CreateTablet for tablet c8a1134a770f44869d9413b601060d90 (DEFAULT_TABLE table=test-table [id=7d14fd53be07426b9e2bed52d1461990]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 06:27:18.910593 4901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:27:18.905690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55920 (local address 127.21.12.130:42265)
0806 06:27:18.905877 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0806 06:27:18.905880 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 06:27:18.906115 (+ 235us) server_negotiation.cc:407] Connection header received
0806 06:27:18.906169 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:27:18.906172 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:27:18.906188 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:27:18.906269 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 06:27:18.906423 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:27:18.907548 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:27:18.908828 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:27:18.909511 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:27:18.909578 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:27:18.910410 (+ 832us) server_negotiation.cc:1035] Waiting for connection context
0806 06:27:18.910462 (+ 52us) server_negotiation.cc:299] Negotiation successful
0806 06:27:18.910508 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":88,"threads_started":1}
I20260806 06:27:18.910681 4386 tablet_service.cc:1467] Processing CreateTablet for tablet c8a1134a770f44869d9413b601060d90 (DEFAULT_TABLE table=test-table [id=7d14fd53be07426b9e2bed52d1461990]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 06:27:18.910837 4794 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c8a1134a770f44869d9413b601060d90. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 06:27:18.911031 4386 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c8a1134a770f44869d9413b601060d90. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 06:27:18.913316 4902 tablet_bootstrap.cc:492] T c8a1134a770f44869d9413b601060d90 P 819c874b0d9e4d13a3c8fb593340484e: Bootstrap starting.
I20260806 06:27:18.913884 4594 tablet_service.cc:1467] Processing CreateTablet for tablet c8a1134a770f44869d9413b601060d90 (DEFAULT_TABLE table=test-table [id=7d14fd53be07426b9e2bed52d1461990]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 06:27:18.914090 4902 tablet_bootstrap.cc:654] T c8a1134a770f44869d9413b601060d90 P 819c874b0d9e4d13a3c8fb593340484e: Neither blocks nor log segments found. Creating new log.
I20260806 06:27:18.914242 4594 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c8a1134a770f44869d9413b601060d90. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 06:27:18.914753 4902 log.cc:824] T c8a1134a770f44869d9413b601060d90 P 819c874b0d9e4d13a3c8fb593340484e: Log is configured to *not* fsync() on all Append() calls
I20260806 06:27:18.915544 4902 tablet_bootstrap.cc:492] T c8a1134a770f44869d9413b601060d90 P 819c874b0d9e4d13a3c8fb593340484e: No bootstrap required, opened a new log
I20260806 06:27:18.915627 4902 ts_tablet_manager.cc:1397] T c8a1134a770f44869d9413b601060d90 P 819c874b0d9e4d13a3c8fb593340484e: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260806 06:27:18.917549 4907 tablet_bootstrap.cc:492] T c8a1134a770f44869d9413b601060d90 P 3d853f8c7420488b895f2ec644e36ee6: Bootstrap starting.
I20260806 06:27:18.917670 4902 raft_consensus.cc:348] T c8a1134a770f44869d9413b601060d90 P 819c874b0d9e4d13a3c8fb593340484e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819c874b0d9e4d13a3c8fb593340484e" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 43901 } } peers { permanent_uuid: "3d853f8c7420488b895f2ec644e36ee6" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 42265 } } peers { permanent_uuid: "7c09c22b0380440ba8ca1759a24c59f2" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 46441 } }
I20260806 06:27:18.917846 4902 raft_consensus.cc:374] T c8a1134a770f44869d9413b601060d90 P 819c874b0d9e4d13a3c8fb593340484e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 06:27:18.917899 4902 raft_consensus.cc:729] T c8a1134a770f44869d9413b601060d90 P 819c874b0d9e4d13a3c8fb593340484e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 819c874b0d9e4d13a3c8fb593340484e, State: Initialized, Role: FOLLOWER
I20260806 06:27:18.918040 4902 consensus_queue.cc:260] T c8a1134a770f44869d9413b601060d90 P 819c874b0d9e4d13a3c8fb593340484e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819c874b0d9e4d13a3c8fb593340484e" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 43901 } } peers { permanent_uuid: "3d853f8c7420488b895f2ec644e36ee6" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 42265 } } peers { permanent_uuid: "7c09c22b0380440ba8ca1759a24c59f2" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 46441 } }
I20260806 06:27:18.918272 4908 tablet_bootstrap.cc:492] T c8a1134a770f44869d9413b601060d90 P 7c09c22b0380440ba8ca1759a24c59f2: Bootstrap starting.
I20260806 06:27:18.918339 4907 tablet_bootstrap.cc:654] T c8a1134a770f44869d9413b601060d90 P 3d853f8c7420488b895f2ec644e36ee6: Neither blocks nor log segments found. Creating new log.
I20260806 06:27:18.918385 4902 ts_tablet_manager.cc:1428] T c8a1134a770f44869d9413b601060d90 P 819c874b0d9e4d13a3c8fb593340484e: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260806 06:27:18.918522 4886 heartbeater.cc:503] Master 127.21.12.190:40289 was elected leader, sending a full tablet report...
I20260806 06:27:18.918882 4907 log.cc:824] T c8a1134a770f44869d9413b601060d90 P 3d853f8c7420488b895f2ec644e36ee6: Log is configured to *not* fsync() on all Append() calls
I20260806 06:27:18.918995 4908 tablet_bootstrap.cc:654] T c8a1134a770f44869d9413b601060d90 P 7c09c22b0380440ba8ca1759a24c59f2: Neither blocks nor log segments found. Creating new log.
I20260806 06:27:18.919611 4908 log.cc:824] T c8a1134a770f44869d9413b601060d90 P 7c09c22b0380440ba8ca1759a24c59f2: Log is configured to *not* fsync() on all Append() calls
I20260806 06:27:18.919783 4907 tablet_bootstrap.cc:492] T c8a1134a770f44869d9413b601060d90 P 3d853f8c7420488b895f2ec644e36ee6: No bootstrap required, opened a new log
I20260806 06:27:18.919898 4907 ts_tablet_manager.cc:1397] T c8a1134a770f44869d9413b601060d90 P 3d853f8c7420488b895f2ec644e36ee6: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260806 06:27:18.920303 4908 tablet_bootstrap.cc:492] T c8a1134a770f44869d9413b601060d90 P 7c09c22b0380440ba8ca1759a24c59f2: No bootstrap required, opened a new log
I20260806 06:27:18.920390 4908 ts_tablet_manager.cc:1397] T c8a1134a770f44869d9413b601060d90 P 7c09c22b0380440ba8ca1759a24c59f2: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260806 06:27:18.921350 4907 raft_consensus.cc:348] T c8a1134a770f44869d9413b601060d90 P 3d853f8c7420488b895f2ec644e36ee6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819c874b0d9e4d13a3c8fb593340484e" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 43901 } } peers { permanent_uuid: "3d853f8c7420488b895f2ec644e36ee6" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 42265 } } peers { permanent_uuid: "7c09c22b0380440ba8ca1759a24c59f2" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 46441 } }
I20260806 06:27:18.921538 4907 raft_consensus.cc:374] T c8a1134a770f44869d9413b601060d90 P 3d853f8c7420488b895f2ec644e36ee6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 06:27:18.921566 4907 raft_consensus.cc:729] T c8a1134a770f44869d9413b601060d90 P 3d853f8c7420488b895f2ec644e36ee6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d853f8c7420488b895f2ec644e36ee6, State: Initialized, Role: FOLLOWER
I20260806 06:27:18.921665 4907 consensus_queue.cc:260] T c8a1134a770f44869d9413b601060d90 P 3d853f8c7420488b895f2ec644e36ee6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819c874b0d9e4d13a3c8fb593340484e" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 43901 } } peers { permanent_uuid: "3d853f8c7420488b895f2ec644e36ee6" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 42265 } } peers { permanent_uuid: "7c09c22b0380440ba8ca1759a24c59f2" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 46441 } }
I20260806 06:27:18.921921 4907 ts_tablet_manager.cc:1428] T c8a1134a770f44869d9413b601060d90 P 3d853f8c7420488b895f2ec644e36ee6: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260806 06:27:18.921949 4684 heartbeater.cc:503] Master 127.21.12.190:40289 was elected leader, sending a full tablet report...
I20260806 06:27:18.922079 4908 raft_consensus.cc:348] T c8a1134a770f44869d9413b601060d90 P 7c09c22b0380440ba8ca1759a24c59f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819c874b0d9e4d13a3c8fb593340484e" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 43901 } } peers { permanent_uuid: "3d853f8c7420488b895f2ec644e36ee6" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 42265 } } peers { permanent_uuid: "7c09c22b0380440ba8ca1759a24c59f2" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 46441 } }
I20260806 06:27:18.922238 4908 raft_consensus.cc:374] T c8a1134a770f44869d9413b601060d90 P 7c09c22b0380440ba8ca1759a24c59f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 06:27:18.922286 4908 raft_consensus.cc:729] T c8a1134a770f44869d9413b601060d90 P 7c09c22b0380440ba8ca1759a24c59f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c09c22b0380440ba8ca1759a24c59f2, State: Initialized, Role: FOLLOWER
I20260806 06:27:18.922431 4908 consensus_queue.cc:260] T c8a1134a770f44869d9413b601060d90 P 7c09c22b0380440ba8ca1759a24c59f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819c874b0d9e4d13a3c8fb593340484e" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 43901 } } peers { permanent_uuid: "3d853f8c7420488b895f2ec644e36ee6" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 42265 } } peers { permanent_uuid: "7c09c22b0380440ba8ca1759a24c59f2" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 46441 } }
I20260806 06:27:18.922734 4908 ts_tablet_manager.cc:1428] T c8a1134a770f44869d9413b601060d90 P 7c09c22b0380440ba8ca1759a24c59f2: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260806 06:27:18.922783 4480 heartbeater.cc:503] Master 127.21.12.190:40289 was elected leader, sending a full tablet report...
W20260806 06:27:18.974722 4686 tablet.cc:2369] T c8a1134a770f44869d9413b601060d90 P 3d853f8c7420488b895f2ec644e36ee6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 06:27:19.050531 4910 raft_consensus.cc:482] T c8a1134a770f44869d9413b601060d90 P 819c874b0d9e4d13a3c8fb593340484e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 06:27:19.050662 4910 raft_consensus.cc:504] T c8a1134a770f44869d9413b601060d90 P 819c874b0d9e4d13a3c8fb593340484e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819c874b0d9e4d13a3c8fb593340484e" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 43901 } } peers { permanent_uuid: "3d853f8c7420488b895f2ec644e36ee6" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 42265 } } peers { permanent_uuid: "7c09c22b0380440ba8ca1759a24c59f2" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 46441 } }
I20260806 06:27:19.050995 4910 leader_election.cc:288] T c8a1134a770f44869d9413b601060d90 P 819c874b0d9e4d13a3c8fb593340484e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3d853f8c7420488b895f2ec644e36ee6 (127.21.12.130:42265), 7c09c22b0380440ba8ca1759a24c59f2 (127.21.12.129:46441)
I20260806 06:27:19.053764 4888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:27:19.051150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.130:42265 (local address 127.21.12.131:35467)
0806 06:27:19.051211 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0806 06:27:19.051225 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 06:27:19.051291 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:27:19.051493 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:27:19.051495 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:27:19.051497 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 06:27:19.051580 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:27:19.051582 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:27:19.052231 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:27:19.052232 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:27:19.053005 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:27:19.053007 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:27:19.053532 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:27:19.053533 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:27:19.053607 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:27:19.053610 (+ 3us) client_negotiation.cc:769] Sending connection context
0806 06:27:19.053655 (+ 45us) client_negotiation.cc:240] Negotiation successful
0806 06:27:19.053687 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260806 06:27:19.053788 4901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:27:19.051281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.131:35467 (local address 127.21.12.130:42265)
0806 06:27:19.051328 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0806 06:27:19.051330 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:27:19.051339 (+ 9us) server_negotiation.cc:407] Connection header received
0806 06:27:19.051369 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:27:19.051370 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:27:19.051383 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:27:19.051439 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 06:27:19.051661 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:27:19.052154 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:27:19.053096 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:27:19.053458 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:27:19.053519 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:27:19.053528 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0806 06:27:19.053665 (+ 137us) server_negotiation.cc:299] Negotiation successful
0806 06:27:19.053703 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260806 06:27:19.053828 4900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:27:19.051264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.131:49713 (local address 127.21.12.129:46441)
0806 06:27:19.051310 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0806 06:27:19.051312 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:27:19.051489 (+ 177us) server_negotiation.cc:407] Connection header received
0806 06:27:19.051522 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:27:19.051523 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:27:19.051536 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:27:19.051594 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 06:27:19.051742 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:27:19.052192 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:27:19.053090 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:27:19.053458 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:27:19.053570 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:27:19.053578 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0806 06:27:19.053730 (+ 152us) server_negotiation.cc:299] Negotiation successful
0806 06:27:19.053766 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260806 06:27:19.053884 4915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:27:19.051157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.129:46441 (local address 127.21.12.131:49713)
0806 06:27:19.051358 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0806 06:27:19.051366 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0806 06:27:19.051431 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:27:19.051602 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:27:19.051603 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:27:19.051604 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 06:27:19.051679 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:27:19.051681 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:27:19.052264 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:27:19.052264 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:27:19.053006 (+ 742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:27:19.053008 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:27:19.053576 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:27:19.053578 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:27:19.053647 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:27:19.053651 (+ 4us) client_negotiation.cc:769] Sending connection context
0806 06:27:19.053714 (+ 63us) client_negotiation.cc:240] Negotiation successful
0806 06:27:19.053751 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187,"mutex_wait_us":55,"thread_start_us":89,"threads_started":1}
I20260806 06:27:19.054169 4618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8a1134a770f44869d9413b601060d90" candidate_uuid: "819c874b0d9e4d13a3c8fb593340484e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d853f8c7420488b895f2ec644e36ee6" is_pre_election: true
I20260806 06:27:19.054243 4422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8a1134a770f44869d9413b601060d90" candidate_uuid: "819c874b0d9e4d13a3c8fb593340484e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c09c22b0380440ba8ca1759a24c59f2" is_pre_election: true
I20260806 06:27:19.054364 4618 raft_consensus.cc:2454] T c8a1134a770f44869d9413b601060d90 P 3d853f8c7420488b895f2ec644e36ee6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 819c874b0d9e4d13a3c8fb593340484e in term 0.
I20260806 06:27:19.054425 4422 raft_consensus.cc:2454] T c8a1134a770f44869d9413b601060d90 P 7c09c22b0380440ba8ca1759a24c59f2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 819c874b0d9e4d13a3c8fb593340484e in term 0.
I20260806 06:27:19.054630 4728 leader_election.cc:302] T c8a1134a770f44869d9413b601060d90 P 819c874b0d9e4d13a3c8fb593340484e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d853f8c7420488b895f2ec644e36ee6, 819c874b0d9e4d13a3c8fb593340484e; no voters:
I20260806 06:27:19.054792 4910 raft_consensus.cc:2781] T c8a1134a770f44869d9413b601060d90 P 819c874b0d9e4d13a3c8fb593340484e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 06:27:19.054850 4910 raft_consensus.cc:482] T c8a1134a770f44869d9413b601060d90 P 819c874b0d9e4d13a3c8fb593340484e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 06:27:19.054875 4910 raft_consensus.cc:3037] T c8a1134a770f44869d9413b601060d90 P 819c874b0d9e4d13a3c8fb593340484e [term 0 FOLLOWER]: Advancing to term 1
I20260806 06:27:19.055588 4910 raft_consensus.cc:504] T c8a1134a770f44869d9413b601060d90 P 819c874b0d9e4d13a3c8fb593340484e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819c874b0d9e4d13a3c8fb593340484e" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 43901 } } peers { permanent_uuid: "3d853f8c7420488b895f2ec644e36ee6" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 42265 } } peers { permanent_uuid: "7c09c22b0380440ba8ca1759a24c59f2" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 46441 } }
I20260806 06:27:19.055707 4910 leader_election.cc:288] T c8a1134a770f44869d9413b601060d90 P 819c874b0d9e4d13a3c8fb593340484e [CANDIDATE]: Term 1 election: Requested vote from peers 3d853f8c7420488b895f2ec644e36ee6 (127.21.12.130:42265), 7c09c22b0380440ba8ca1759a24c59f2 (127.21.12.129:46441)
I20260806 06:27:19.055878 4422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8a1134a770f44869d9413b601060d90" candidate_uuid: "819c874b0d9e4d13a3c8fb593340484e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c09c22b0380440ba8ca1759a24c59f2"
I20260806 06:27:19.055889 4618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8a1134a770f44869d9413b601060d90" candidate_uuid: "819c874b0d9e4d13a3c8fb593340484e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d853f8c7420488b895f2ec644e36ee6"
I20260806 06:27:19.055949 4422 raft_consensus.cc:3037] T c8a1134a770f44869d9413b601060d90 P 7c09c22b0380440ba8ca1759a24c59f2 [term 0 FOLLOWER]: Advancing to term 1
I20260806 06:27:19.055960 4618 raft_consensus.cc:3037] T c8a1134a770f44869d9413b601060d90 P 3d853f8c7420488b895f2ec644e36ee6 [term 0 FOLLOWER]: Advancing to term 1
I20260806 06:27:19.056661 4422 raft_consensus.cc:2454] T c8a1134a770f44869d9413b601060d90 P 7c09c22b0380440ba8ca1759a24c59f2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 819c874b0d9e4d13a3c8fb593340484e in term 1.
I20260806 06:27:19.056752 4618 raft_consensus.cc:2454] T c8a1134a770f44869d9413b601060d90 P 3d853f8c7420488b895f2ec644e36ee6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 819c874b0d9e4d13a3c8fb593340484e in term 1.
I20260806 06:27:19.056831 4724 leader_election.cc:302] T c8a1134a770f44869d9413b601060d90 P 819c874b0d9e4d13a3c8fb593340484e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7c09c22b0380440ba8ca1759a24c59f2, 819c874b0d9e4d13a3c8fb593340484e; no voters:
I20260806 06:27:19.056931 4910 raft_consensus.cc:2781] T c8a1134a770f44869d9413b601060d90 P 819c874b0d9e4d13a3c8fb593340484e [term 1 FOLLOWER]: Leader election won for term 1
I20260806 06:27:19.057088 4910 raft_consensus.cc:686] T c8a1134a770f44869d9413b601060d90 P 819c874b0d9e4d13a3c8fb593340484e [term 1 LEADER]: Becoming Leader. State: Replica: 819c874b0d9e4d13a3c8fb593340484e, State: Running, Role: LEADER
I20260806 06:27:19.057174 4910 consensus_queue.cc:237] T c8a1134a770f44869d9413b601060d90 P 819c874b0d9e4d13a3c8fb593340484e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819c874b0d9e4d13a3c8fb593340484e" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 43901 } } peers { permanent_uuid: "3d853f8c7420488b895f2ec644e36ee6" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 42265 } } peers { permanent_uuid: "7c09c22b0380440ba8ca1759a24c59f2" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 46441 } }
I20260806 06:27:19.057981 4193 catalog_manager.cc:5495] T c8a1134a770f44869d9413b601060d90 P 819c874b0d9e4d13a3c8fb593340484e reported cstate change: term changed from 0 to 1, leader changed from <none> to 819c874b0d9e4d13a3c8fb593340484e (127.21.12.131). New cstate: current_term: 1 leader_uuid: "819c874b0d9e4d13a3c8fb593340484e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "819c874b0d9e4d13a3c8fb593340484e" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 43901 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3d853f8c7420488b895f2ec644e36ee6" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 42265 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7c09c22b0380440ba8ca1759a24c59f2" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 46441 } health_report { overall_health: UNKNOWN } } }
W20260806 06:27:19.077178 4482 tablet.cc:2369] T c8a1134a770f44869d9413b601060d90 P 7c09c22b0380440ba8ca1759a24c59f2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 06:27:19.096135 4898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:27:19.093943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55102 (local address 127.21.12.131:43901)
0806 06:27:19.094032 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0806 06:27:19.094035 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 06:27:19.094044 (+ 9us) server_negotiation.cc:407] Connection header received
0806 06:27:19.094087 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:27:19.094087 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:27:19.094111 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:27:19.094177 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 06:27:19.094313 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:27:19.094810 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:27:19.095518 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:27:19.095702 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:27:19.095751 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:27:19.095852 (+ 101us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 06:27:19.095947 (+ 95us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 06:27:19.096010 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0806 06:27:19.096021 (+ 11us) server_negotiation.cc:299] Negotiation successful
0806 06:27:19.096036 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260806 06:27:19.097910 4422 raft_consensus.cc:1261] T c8a1134a770f44869d9413b601060d90 P 7c09c22b0380440ba8ca1759a24c59f2 [term 1 FOLLOWER]: Refusing update from remote peer 819c874b0d9e4d13a3c8fb593340484e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 06:27:19.097929 4618 raft_consensus.cc:1261] T c8a1134a770f44869d9413b601060d90 P 3d853f8c7420488b895f2ec644e36ee6 [term 1 FOLLOWER]: Refusing update from remote peer 819c874b0d9e4d13a3c8fb593340484e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 06:27:19.098238 4910 consensus_queue.cc:1035] T c8a1134a770f44869d9413b601060d90 P 819c874b0d9e4d13a3c8fb593340484e [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d853f8c7420488b895f2ec644e36ee6" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 42265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 06:27:19.098376 4916 consensus_queue.cc:1035] T c8a1134a770f44869d9413b601060d90 P 819c874b0d9e4d13a3c8fb593340484e [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c09c22b0380440ba8ca1759a24c59f2" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 46441 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 06:27:19.100919 4929 mvcc.cc:205] Tried to move back new op lower bound from 7315446329742368768 to 7315446329578860544. Current Snapshot: MvccSnapshot[applied={T|T < 7315446329742368768}]
I20260806 06:27:19.101133 4926 mvcc.cc:205] Tried to move back new op lower bound from 7315446329742368768 to 7315446329578860544. Current Snapshot: MvccSnapshot[applied={T|T < 7315446329742368768}]
I20260806 06:27:19.103008 4191 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60694:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 06:27:19.103088 4191 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60694:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 06:27:19.103762 4191 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 15963a3d7a204faaa545504b24a5ebfc: Sending DeleteTablet for 3 replicas of tablet c8a1134a770f44869d9413b601060d90
I20260806 06:27:19.104079 4386 tablet_service.cc:1514] Processing DeleteTablet for tablet c8a1134a770f44869d9413b601060d90 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 06:27:19 UTC) from {username='kudu', principal='kudu/127.21.12.190@KRBTEST.COM'} at 127.0.0.1:38628
I20260806 06:27:19.104102 4594 tablet_service.cc:1514] Processing DeleteTablet for tablet c8a1134a770f44869d9413b601060d90 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 06:27:19 UTC) from {username='kudu', principal='kudu/127.21.12.190@KRBTEST.COM'} at 127.0.0.1:55920
I20260806 06:27:19.104302 4945 tablet_replica.cc:330] T c8a1134a770f44869d9413b601060d90 P 3d853f8c7420488b895f2ec644e36ee6: stopping tablet replica
I20260806 06:27:19.104328 21554 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHhLhT/build/release/bin/kudu with pid 4293
I20260806 06:27:19.104444 4794 tablet_service.cc:1514] Processing DeleteTablet for tablet c8a1134a770f44869d9413b601060d90 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 06:27:19 UTC) from {username='kudu', principal='kudu/127.21.12.190@KRBTEST.COM'} at 127.0.0.1:55092
I20260806 06:27:19.104629 4945 raft_consensus.cc:2229] T c8a1134a770f44869d9413b601060d90 P 3d853f8c7420488b895f2ec644e36ee6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 06:27:19.104801 4945 raft_consensus.cc:2258] T c8a1134a770f44869d9413b601060d90 P 3d853f8c7420488b895f2ec644e36ee6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 06:27:19.114755 4946 tablet_replica.cc:330] T c8a1134a770f44869d9413b601060d90 P 819c874b0d9e4d13a3c8fb593340484e: stopping tablet replica
I20260806 06:27:19.114877 4946 raft_consensus.cc:2229] T c8a1134a770f44869d9413b601060d90 P 819c874b0d9e4d13a3c8fb593340484e [term 1 LEADER]: Raft consensus shutting down.
I20260806 06:27:19.114977 4946 raft_consensus.cc:2258] T c8a1134a770f44869d9413b601060d90 P 819c874b0d9e4d13a3c8fb593340484e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 06:27:19.115053 4945 ts_tablet_manager.cc:1905] T c8a1134a770f44869d9413b601060d90 P 3d853f8c7420488b895f2ec644e36ee6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 06:27:19.115401 4946 ts_tablet_manager.cc:1905] T c8a1134a770f44869d9413b601060d90 P 819c874b0d9e4d13a3c8fb593340484e: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260806 06:27:19.116870 4173 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.12.129:46441 (error 108)
I20260806 06:27:19.116989 4945 ts_tablet_manager.cc:1918] T c8a1134a770f44869d9413b601060d90 P 3d853f8c7420488b895f2ec644e36ee6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 06:27:19.117065 21554 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHhLhT/build/release/bin/kudu with pid 4486
I20260806 06:27:19.125275 4897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:27:19.125116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.129:46441 (local address 127.0.0.1:38638)
0806 06:27:19.125180 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0806 06:27:19.125236 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.129:46441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260806 06:27:19.125403 4173 catalog_manager.cc:4584] TS 7c09c22b0380440ba8ca1759a24c59f2 (127.21.12.129:46441): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c8a1134a770f44869d9413b601060d90: Network error: Client connection negotiation failed: client connection to 127.21.12.129:46441: connect: Connection refused (error 111)
I20260806 06:27:19.125507 4946 ts_tablet_manager.cc:1918] T c8a1134a770f44869d9413b601060d90 P 819c874b0d9e4d13a3c8fb593340484e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 06:27:19.125561 4946 log.cc:1196] T c8a1134a770f44869d9413b601060d90 P 819c874b0d9e4d13a3c8fb593340484e: Deleting WAL directory at /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785997581049705-21554-0/minicluster-data/ts-2/wal/wals/c8a1134a770f44869d9413b601060d90
I20260806 06:27:19.125770 4946 ts_tablet_manager.cc:1939] T c8a1134a770f44869d9413b601060d90 P 819c874b0d9e4d13a3c8fb593340484e: Deleting consensus metadata
I20260806 06:27:19.128075 4176 catalog_manager.cc:4841] TS 819c874b0d9e4d13a3c8fb593340484e (127.21.12.131:43901): tablet c8a1134a770f44869d9413b601060d90 (table test-table [id=7d14fd53be07426b9e2bed52d1461990]) successfully deleted
I20260806 06:27:19.131628 21554 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHhLhT/build/release/bin/kudu with pid 4692
I20260806 06:27:19.132979 4897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:27:19.131727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.130:42265 (local address 127.0.0.1:55930)
0806 06:27:19.132911 (+ 1184us) negotiation.cc:107] Waiting for socket to connect
0806 06:27:19.132948 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.130:42265: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1162}
I20260806 06:27:19.143266 21554 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHhLhT/build/release/bin/kudu with pid 4150
2026-08-06T06:27:19Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2118 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785997581049705-21554-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 06:27:19 dist-test-slave-4tdj krb5kdc[4955](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 06:27:19 dist-test-slave-4tdj krb5kdc[4955](info): set up 2 sockets
Aug 06 06:27:19 dist-test-slave-4tdj krb5kdc[4955](info): commencing operation
krb5kdc: starting...
W20260806 06:27:20.267097 21554 mini_kdc.cc:121] Time spent starting KDC: real 1.101s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785997581049705-21554-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785997581049705-21554-0/krb5kdc/test-user.keytab.
Aug 06 06:27:20 dist-test-slave-4tdj krb5kdc[4955](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997640, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T06:27:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T06:27:20Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.21.12.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.21.12.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.21.12.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785997581049705-21554-0/krb5kdc/rangeradmin_127.21.12.148@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.21.12.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785997581049705-21554-0/krb5kdc/rangeradmin_127.21.12.148@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.21.12.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.21.12.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.21.12.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785997581049705-21554-0/krb5kdc/rangerlookup_127.21.12.148@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.21.12.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785997581049705-21554-0/krb5kdc/rangerlookup_127.21.12.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.21.12.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.21.12.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.21.12.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785997581049705-21554-0/krb5kdc/HTTP_127.21.12.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.21.12.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785997581049705-21554-0/krb5kdc/HTTP_127.21.12.148@KRBTEST.COM.keytab.
I20260806 06:27:20.418592 21554 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785997581049705-21554-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskFHhLhT/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785997581049705-21554-0/minicluster-data/postgres -l logfile start
2026-08-06 06:27:22.742 UTC [5015] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-06 06:27:22.742 UTC [5015] LOG: listening on IPv4 address "127.21.12.148", port 49301
2026-08-06 06:27:22.745 UTC [5015] LOG: listening on Unix socket "/tmp/.s.PGSQL.49301"
2026-08-06 06:27:22.765 UTC [5020] LOG: database system was shut down at 2026-08-06 06:27:21 UTC
2026-08-06 06:27:22.768 UTC [5015] LOG: database system is ready to accept connections
I20260806 06:27:23.799700 21554 mini_postgres.cc:96] Postgres bound to 49301
2026-08-06 06:27:23.808 UTC [5034] FATAL: database "slave" does not exist
127.21.12.148:49301 - accepting connections
I20260806 06:27:23.809165 21554 mini_ranger.cc:161] Starting Ranger...
I20260806 06:27:23.825111 21554 mini_ranger.cc:84] Created miniranger Postgres user
I20260806 06:27:24.231019 21554 mini_ranger.cc:87] Created ranger Postgres database
I20260806 06:27:24.231102 21554 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785997581049705-21554-0/minicluster-data/ranger-admin
2026-08-06 06:27:24,539 [I] DB FLAVOR :POSTGRES
2026-08-06 06:27:24,540 [I] --------- Verifying Ranger DB connection ---------
2026-08-06 06:27:24,540 [I] Checking connection..
2026-08-06 06:27:24,540 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFHhLhT/build/release/bin/postgresql.jar:/tmp/dist-test-taskFHhLhT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.12.148:49301/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-06 06:27:24,799 [I] Checking connection passed.
2026-08-06 06:27:24,800 [I] --------- Verifying version history table ---------
2026-08-06 06:27:24,800 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFHhLhT/build/release/bin/postgresql.jar:/tmp/dist-test-taskFHhLhT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.12.148:49301/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-06 06:27:25,056 [I] Table x_db_version_h does not exist in database ranger
2026-08-06 06:27:25,056 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFHhLhT/build/release/bin/postgresql.jar:/tmp/dist-test-taskFHhLhT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.12.148:49301/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-06 06:27:25,289 [I] Table x_db_version_h does not exist in database ranger
2026-08-06 06:27:25,290 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-06 06:27:25,290 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFHhLhT/build/release/bin/postgresql.jar:/tmp/dist-test-taskFHhLhT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.12.148:49301/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskFHhLhT/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-06 06:27:25.533 UTC [5140] WARNING: there is no transaction in progress
2026-08-06 06:27:25,541 [I] create_dbversion_catalog.sql file imported successfully
2026-08-06 06:27:25,542 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFHhLhT/build/release/bin/postgresql.jar:/tmp/dist-test-taskFHhLhT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.12.148:49301/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-06 06:27:25,788 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-06 06:27:25,788 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-06 06:27:25,789 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFHhLhT/build/release/bin/postgresql.jar:/tmp/dist-test-taskFHhLhT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.12.148:49301/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-06 06:27:26,042 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFHhLhT/build/release/bin/postgresql.jar:/tmp/dist-test-taskFHhLhT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.12.148:49301/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-06 06:27:26,264 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFHhLhT/build/release/bin/postgresql.jar:/tmp/dist-test-taskFHhLhT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.12.148:49301/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-4tdj.c.gcp-upstream.internal','N') ;"
2026-08-06 06:27:26,502 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFHhLhT/build/release/bin/postgresql.jar:/tmp/dist-test-taskFHhLhT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.12.148:49301/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-06 06:27:26,741 [I] Table x_portal_user does not exist in database ranger
2026-08-06 06:27:26,742 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFHhLhT/build/release/bin/postgresql.jar:/tmp/dist-test-taskFHhLhT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.12.148:49301/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-06 06:27:27,017 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-06 06:27:27,018 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFHhLhT/build/release/bin/postgresql.jar:/tmp/dist-test-taskFHhLhT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.12.148:49301/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-06 06:27:27,257 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFHhLhT/build/release/bin/postgresql.jar:/tmp/dist-test-taskFHhLhT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.12.148:49301/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-06 06:27:27,520 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-06 06:27:27,520 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFHhLhT/build/release/bin/postgresql.jar:/tmp/dist-test-taskFHhLhT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.12.148:49301/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskFHhLhT/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-06 06:27:28.449 UTC [5311] WARNING: there is no transaction in progress
2026-08-06 06:27:28.464 UTC [5311] WARNING: there is no transaction in progress
2026-08-06 06:27:28.480 UTC [5311] WARNING: there is no transaction in progress
2026-08-06 06:27:28.494 UTC [5311] WARNING: there is no transaction in progress
2026-08-06 06:27:28.509 UTC [5311] WARNING: there is no transaction in progress
2026-08-06 06:27:28.525 UTC [5311] WARNING: there is no transaction in progress
2026-08-06 06:27:28.576 UTC [5311] WARNING: there is no transaction in progress
2026-08-06 06:27:28.582 UTC [5311] WARNING: there is no transaction in progress
2026-08-06 06:27:28.595 UTC [5311] WARNING: there is no transaction in progress
2026-08-06 06:27:28.605 UTC [5311] WARNING: there is no transaction in progress
2026-08-06 06:27:28.614 UTC [5311] WARNING: there is no transaction in progress
2026-08-06 06:27:28.626 UTC [5311] WARNING: there is no transaction in progress
2026-08-06 06:27:28.635 UTC [5311] WARNING: there is no transaction in progress
2026-08-06 06:27:28.641 UTC [5311] WARNING: there is no transaction in progress
2026-08-06 06:27:28.647 UTC [5311] WARNING: there is no transaction in progress
2026-08-06 06:27:28.652 UTC [5311] WARNING: there is no transaction in progress
2026-08-06 06:27:29,223 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-06 06:27:29,223 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFHhLhT/build/release/bin/postgresql.jar:/tmp/dist-test-taskFHhLhT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.12.148:49301/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-06 06:27:29,479 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-06 06:27:29,479 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFHhLhT/build/release/bin/postgresql.jar:/tmp/dist-test-taskFHhLhT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.12.148:49301/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-06 06:27:29,741 [I] Table x_portal_user already exists in database 'ranger'
2026-08-06 06:27:29,741 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFHhLhT/build/release/bin/postgresql.jar:/tmp/dist-test-taskFHhLhT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.12.148:49301/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-06 06:27:30,003 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-06 06:27:30,003 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFHhLhT/build/release/bin/postgresql.jar:/tmp/dist-test-taskFHhLhT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.12.148:49301/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-06 06:27:30,243 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFHhLhT/build/release/bin/postgresql.jar:/tmp/dist-test-taskFHhLhT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.12.148:49301/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-06 06:27:30,501 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFHhLhT/build/release/bin/postgresql.jar:/tmp/dist-test-taskFHhLhT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.12.148:49301/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-4tdj.c.gcp-upstream.internal';"
2026-08-06 06:27:30,725 [I] CORE_DB_SCHEMA import status has been updated
2026-08-06 06:27:30,726 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFHhLhT/build/release/bin/postgresql.jar:/tmp/dist-test-taskFHhLhT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.12.148:49301/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-06 06:27:30,961 [I] DB_PATCHES have already been applied
I20260806 06:27:30.967222 21554 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785997581049705-21554-0/minicluster-data/ranger-admin:/tmp/dist-test-taskFHhLhT/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskFHhLhT/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskFHhLhT/build/release/bin/postgresql.jar:/tmp/dist-test-taskFHhLhT/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260806 06:27:30.967283 21554 mini_ranger.cc:193] Using host: 127.21.12.148
I20260806 06:27:30.970988 21554 mini_ranger.cc:227] Ranger admin URL: http://127.21.12.148:44277
Aug 06, 2026 6:27:31 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 06, 2026 6:27:31 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 06, 2026 6:27:31 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785997581049705-21554-0/minicluster-data/ranger-admin/ews/webapp
Aug 06, 2026 6:27:31 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785997581049705-21554-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 06, 2026 6:27:31 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785997581049705-21554-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 06, 2026 6:27:31 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785997581049705-21554-0/minicluster-data/ranger-admin/ews/webapp].
Aug 06, 2026 6:27:31 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260806 06:27:32.203897 21554 mini_ranger.cc:160] Time spent starting Ranger: real 8.395s user 0.000s sys 0.010s
Aug 06, 2026 6:27:51 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 06, 2026 6:27:51 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Aug 06, 2026 6:27:51 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 06, 2026 6:27:51 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 06, 2026 6:27:51 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 06, 2026 6:27:51 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 06, 2026 6:27:51 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 06, 2026 6:27:51 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 06, 2026 6:27:51 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 06, 2026 6:27:51 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 06, 2026 6:27:51 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 06, 2026 6:27:51 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 06, 2026 6:27:51 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 06, 2026 6:27:51 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 06, 2026 6:27:51 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 06, 2026 6:27:51 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 06, 2026 6:27:51 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 06, 2026 6:27:52 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260806 06:27:52.700344 21554 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.21.12.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.21.12.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.21.12.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785997581049705-21554-0/krb5kdc/rangerkms_127.21.12.148@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.21.12.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785997581049705-21554-0/krb5kdc/rangerkms_127.21.12.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.21.12.148@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.21.12.148@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.21.12.148@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785997581049705-21554-0/krb5kdc/HTTP_127.21.12.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.21.12.148@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785997581049705-21554-0/krb5kdc/HTTP_127.21.12.148@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 06 06:27:52 dist-test-slave-4tdj krb5kdc[4955](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997672, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260806 06:27:52.793059 21554 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260806 06:27:52.802906 21554 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260806 06:27:53.611709 21554 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260806 06:27:53.611794 21554 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785997581049705-21554-0/minicluster-data/ranger-kms
I20260806 06:27:53.611810 21554 mini_ranger_kms.cc:227] Using postgres at 127.21.12.148:49301
2026-08-06 06:27:54,311 [I] DB FLAVOR :POSTGRES
2026-08-06 06:27:54,311 [I] --------- Verifying Ranger DB connection ---------
2026-08-06 06:27:54,311 [I] Checking connection
2026-08-06 06:27:54,311 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFHhLhT/build/release/bin/postgresql.jar:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785997581049705-21554-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.12.148:49301/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-06 06:27:54,639 [I] connection success
2026-08-06 06:27:54,639 [I] --------- Verifying Ranger DB tables ---------
2026-08-06 06:27:54,639 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-06 06:27:54,639 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFHhLhT/build/release/bin/postgresql.jar:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785997581049705-21554-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.12.148:49301/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-06 06:27:54,881 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-06 06:27:54,881 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-06 06:27:54,882 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-06 06:27:54,882 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFHhLhT/build/release/bin/postgresql.jar:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785997581049705-21554-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.12.148:49301/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785997581049705-21554-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-06 06:27:55,141 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260806 06:27:55.590719 21554 mini_ranger_kms.cc:310] Created kms service
I20260806 06:27:55.783980 21554 mini_ranger_kms.cc:326] Created kudu user
I20260806 06:27:55.824009 21554 mini_ranger_kms.cc:343] Created rangerkms user
Aug 06 06:27:55 dist-test-slave-4tdj krb5kdc[4955](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997672, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.21.12.148@KRBTEST.COM
I20260806 06:27:56.185490 21554 mini_ranger_kms.cc:384] Added ranger policy
I20260806 06:27:56.185600 21554 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785997581049705-21554-0/minicluster-data/ranger-kms:/tmp/dist-test-taskFHhLhT/build/release/bin/postgresql.jar:/tmp/dist-test-taskFHhLhT/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskFHhLhT/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260806 06:27:56.185621 21554 mini_ranger_kms.cc:240] Using host: 127.21.12.148
I20260806 06:27:56.189441 21554 mini_ranger_kms.cc:276] Ranger KMS PID: 5573
I20260806 06:27:56.189556 21554 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.21.12.148:35907
Aug 06, 2026 6:27:57 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 06, 2026 6:27:57 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 06, 2026 6:27:57 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785997581049705-21554-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 06, 2026 6:27:57 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785997581049705-21554-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 06, 2026 6:27:57 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785997581049705-21554-0/minicluster-data/ranger-kms/ews/webapp].
Aug 06, 2026 6:27:57 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260806 06:27:58.514750 21554 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.722s user 0.009s sys 0.654s
I20260806 06:27:58.514848 21554 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260806 06:27:58.514936 21554 mini_ranger_kms.cc:401] 127.21.12.148:35907/kms/v1/keys
Aug 06 06:27:58 dist-test-slave-4tdj krb5kdc[4955](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1785997678, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.21.12.148@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Aug 06 06:27:59 dist-test-slave-4tdj krb5kdc[4955](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997678, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.21.12.148@KRBTEST.COM for HTTP/127.21.12.148@KRBTEST.COM
Aug 06 06:27:59 dist-test-slave-4tdj krb5kdc[4955](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997678, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.21.12.148@KRBTEST.COM for HTTP/127.21.12.148@KRBTEST.COM
Aug 06 06:28:02 dist-test-slave-4tdj krb5kdc[4955](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997682, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.21.12.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.12.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.12.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.190.keytab.
Entry for principal kudu/127.21.12.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.190.keytab.
I20260806 06:28:02.313197 21554 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHhLhT/build/release/bin/kudu
/tmp/dist-test-taskFHhLhT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785997581049705-21554-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785997581049705-21554-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785997581049705-21554-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785997581049705-21554-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.12.190:39335
--webserver_interface=127.21.12.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.190.keytab
--principal=kudu/127.21.12.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.148:40857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.12.190:39335
--ranger_config_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785997581049705-21554-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.21.12.148:35907/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785997581049705-21554-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785997581049705-21554-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785997581049705-21554-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 06:28:02.455188 5640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 06:28:02.455389 5640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 06:28:02.455410 5640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 06:28:02.456573 5640 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 06:28:02.456620 5640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 06:28:02.456633 5640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 06:28:02.456645 5640 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 06:28:02.456655 5640 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 06:28:02.458499 5640 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.148:40857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785997581049705-21554-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785997581049705-21554-0/minicluster-data/master-0/wal
--ranger_kms_url=127.21.12.148:35907/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.21.12.190:39335
--ranger_config_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785997581049705-21554-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.190.keytab
--principal=kudu/127.21.12.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.12.190:39335
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785997581049705-21554-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.12.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5640
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785997581049705-21554-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 05:11:02 UTC on ccycloud.quasar-hvzbbz.root.comops.site
build id 4569617
I20260806 06:28:02.458735 5640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 06:28:02.458931 5640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260806 06:28:02.465514 5640 server_base.cc:1034] running on GCE node
W20260806 06:28:02.465514 5647 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:28:02.465575 5650 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:28:02.465577 5646 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:28:02.465894 5640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 06:28:02.466142 5640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 06:28:02.467293 5640 hybrid_clock.cc:648] HybridClock initialized: now 1785997682467277 us; error 34 us; skew 500 ppm
Aug 06 06:28:02 dist-test-slave-4tdj krb5kdc[4955](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997682, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 06:28:02.470144 5640 init.cc:377] Logged in from keytab as kudu/127.21.12.190@KRBTEST.COM (short username kudu)
I20260806 06:28:02.470566 5640 webserver.cc:460] Webserver started at http://127.21.12.190:45643/ using document root <none> and password file <none>
I20260806 06:28:02.470769 5640 fs_manager.cc:359] Metadata directory not provided
I20260806 06:28:02.470809 5640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 06:28:02.470928 5640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 06 06:28:02 dist-test-slave-4tdj krb5kdc[4955](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997682, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.190@KRBTEST.COM for HTTP/127.21.12.148@KRBTEST.COM
I20260806 06:28:02.521039 5640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785997581049705-21554-0/minicluster-data/master-0/data/instance:
uuid: "62571dafbdd1430ab9a29c75964ce261"
format_stamp: "Formatted at 2026-08-06 06:28:02 on dist-test-slave-4tdj"
server_key: "96e3e00f5f633fb9c1a7ddee05115f87"
server_key_iv: "76bc153c8592db7af352cfdd602abdfd"
server_key_version: "kuduclusterkey@0"
I20260806 06:28:02.521575 5640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785997581049705-21554-0/minicluster-data/master-0/wal/instance:
uuid: "62571dafbdd1430ab9a29c75964ce261"
format_stamp: "Formatted at 2026-08-06 06:28:02 on dist-test-slave-4tdj"
server_key: "96e3e00f5f633fb9c1a7ddee05115f87"
server_key_iv: "76bc153c8592db7af352cfdd602abdfd"
server_key_version: "kuduclusterkey@0"
I20260806 06:28:02.523861 5640 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.004s
I20260806 06:28:02.538950 5664 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 06:28:02.539139 5640 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260806 06:28:02.539235 5640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785997581049705-21554-0/minicluster-data/master-0/data,/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785997581049705-21554-0/minicluster-data/master-0/wal
uuid: "62571dafbdd1430ab9a29c75964ce261"
format_stamp: "Formatted at 2026-08-06 06:28:02 on dist-test-slave-4tdj"
server_key: "96e3e00f5f633fb9c1a7ddee05115f87"
server_key_iv: "76bc153c8592db7af352cfdd602abdfd"
server_key_version: "kuduclusterkey@0"
I20260806 06:28:02.539326 5640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785997581049705-21554-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785997581049705-21554-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785997581049705-21554-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 06:28:02.559437 5640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 06:28:02.561779 5640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 06:28:02.562021 5640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 06:28:02.577219 5640 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.190:39335
I20260806 06:28:02.577314 5752 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.190:39335 every 8 connection(s)
I20260806 06:28:02.577906 5640 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785997581049705-21554-0/minicluster-data/master-0/data/info.pb
I20260806 06:28:02.579025 5754 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 06:28:02.580533 21554 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHhLhT/build/release/bin/kudu as pid 5640
I20260806 06:28:02.580649 21554 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785997581049705-21554-0/minicluster-data/master-0/wal/instance
I20260806 06:28:02.580847 21554 external_mini_cluster.cc:1428] Setting key bcc9ca2575491593eb8df7c42f3b75ad
I20260806 06:28:02.583679 5754 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 62571dafbdd1430ab9a29c75964ce261: Bootstrap starting.
I20260806 06:28:02.584832 5754 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 62571dafbdd1430ab9a29c75964ce261: Neither blocks nor log segments found. Creating new log.
I20260806 06:28:02.585732 5754 log.cc:824] T 00000000000000000000000000000000 P 62571dafbdd1430ab9a29c75964ce261: Log is configured to *not* fsync() on all Append() calls
I20260806 06:28:02.586603 5754 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 62571dafbdd1430ab9a29c75964ce261: No bootstrap required, opened a new log
Aug 06 06:28:02 dist-test-slave-4tdj krb5kdc[4955](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997682, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.12.190@KRBTEST.COM
I20260806 06:28:02.590674 5754 raft_consensus.cc:348] T 00000000000000000000000000000000 P 62571dafbdd1430ab9a29c75964ce261 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62571dafbdd1430ab9a29c75964ce261" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 39335 } }
I20260806 06:28:02.590837 5754 raft_consensus.cc:374] T 00000000000000000000000000000000 P 62571dafbdd1430ab9a29c75964ce261 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 06:28:02.590901 5754 raft_consensus.cc:729] T 00000000000000000000000000000000 P 62571dafbdd1430ab9a29c75964ce261 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62571dafbdd1430ab9a29c75964ce261, State: Initialized, Role: FOLLOWER
I20260806 06:28:02.591040 5754 consensus_queue.cc:260] T 00000000000000000000000000000000 P 62571dafbdd1430ab9a29c75964ce261 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62571dafbdd1430ab9a29c75964ce261" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 39335 } }
I20260806 06:28:02.591135 5754 raft_consensus.cc:388] T 00000000000000000000000000000000 P 62571dafbdd1430ab9a29c75964ce261 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 06:28:02.591179 5754 raft_consensus.cc:482] T 00000000000000000000000000000000 P 62571dafbdd1430ab9a29c75964ce261 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 06:28:02.591221 5754 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 62571dafbdd1430ab9a29c75964ce261 [term 0 FOLLOWER]: Advancing to term 1
I20260806 06:28:02.592051 5754 raft_consensus.cc:504] T 00000000000000000000000000000000 P 62571dafbdd1430ab9a29c75964ce261 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62571dafbdd1430ab9a29c75964ce261" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 39335 } }
I20260806 06:28:02.592187 5754 leader_election.cc:302] T 00000000000000000000000000000000 P 62571dafbdd1430ab9a29c75964ce261 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 62571dafbdd1430ab9a29c75964ce261; no voters:
I20260806 06:28:02.592437 5754 leader_election.cc:288] T 00000000000000000000000000000000 P 62571dafbdd1430ab9a29c75964ce261 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 06:28:02.592546 5761 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 62571dafbdd1430ab9a29c75964ce261 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 06:28:02.592790 5761 raft_consensus.cc:686] T 00000000000000000000000000000000 P 62571dafbdd1430ab9a29c75964ce261 [term 1 LEADER]: Becoming Leader. State: Replica: 62571dafbdd1430ab9a29c75964ce261, State: Running, Role: LEADER
I20260806 06:28:02.592943 5761 consensus_queue.cc:237] T 00000000000000000000000000000000 P 62571dafbdd1430ab9a29c75964ce261 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62571dafbdd1430ab9a29c75964ce261" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 39335 } }
I20260806 06:28:02.593436 5761 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62571dafbdd1430ab9a29c75964ce261 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 62571dafbdd1430ab9a29c75964ce261. Latest consensus state: current_term: 1 leader_uuid: "62571dafbdd1430ab9a29c75964ce261" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62571dafbdd1430ab9a29c75964ce261" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 39335 } } }
I20260806 06:28:02.593585 5764 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62571dafbdd1430ab9a29c75964ce261 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "62571dafbdd1430ab9a29c75964ce261" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62571dafbdd1430ab9a29c75964ce261" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 39335 } } }
I20260806 06:28:02.593705 5764 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62571dafbdd1430ab9a29c75964ce261 [sys.catalog]: This master's current role is: LEADER
I20260806 06:28:02.593586 5761 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62571dafbdd1430ab9a29c75964ce261 [sys.catalog]: This master's current role is: LEADER
I20260806 06:28:02.594086 5766 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 06:28:02.594796 5766 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 06:28:02.595480 5758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:28:02.581508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54580 (local address 127.21.12.190:39335)
0806 06:28:02.581828 (+ 320us) server_negotiation.cc:206] Beginning negotiation
0806 06:28:02.581838 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 06:28:02.581885 (+ 47us) server_negotiation.cc:407] Connection header received
0806 06:28:02.582641 (+ 756us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:28:02.582652 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:28:02.582723 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:28:02.582880 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:28:02.583867 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:28:02.585436 (+ 1569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:28:02.586584 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:28:02.587564 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:28:02.587694 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:28:02.590676 (+ 2982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:28:02.590681 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:28:02.590681 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 06:28:02.590695 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:28:02.593386 (+ 2691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:28:02.593800 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:28:02.593803 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:28:02.593804 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:28:02.593906 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:28:02.594127 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:28:02.594127 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:28:02.594128 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:28:02.594454 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:28:02.594664 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0806 06:28:02.595114 (+ 450us) server_negotiation.cc:299] Negotiation successful
0806 06:28:02.595236 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":255,"thread_start_us":158,"threads_started":1}
I20260806 06:28:02.592845 5754 sys_catalog.cc:564] T 00000000000000000000000000000000 P 62571dafbdd1430ab9a29c75964ce261 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 06:28:02.598213 5766 catalog_manager.cc:1347] Generated new cluster ID: e8fe96b9545b4292a53e0c00750d034e
I20260806 06:28:02.598268 5766 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 06:28:02.598603 5754 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785997581049705-21554-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260806 06:28:02.618371 5766 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 06:28:02.619271 5766 catalog_manager.cc:1504] Loading token signing keys...
I20260806 06:28:02.623713 5766 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 62571dafbdd1430ab9a29c75964ce261: Generated new TSK 0
I20260806 06:28:02.623878 5766 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.12.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.12.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.12.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.129.keytab.
Entry for principal kudu/127.21.12.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.129.keytab.
I20260806 06:28:03.566867 21554 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHhLhT/build/release/bin/kudu
/tmp/dist-test-taskFHhLhT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785997581049705-21554-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785997581049705-21554-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785997581049705-21554-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785997581049705-21554-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.21.12.148:35907/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.12.129:0
--local_ip_for_outbound_sockets=127.21.12.129
--webserver_interface=127.21.12.129
--webserver_port=0
--tserver_master_addrs=127.21.12.190:39335
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.129.keytab
--principal=kudu/127.21.12.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.148:40857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785997581049705-21554-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785997581049705-21554-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785997581049705-21554-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 06:28:03.705513 5819 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 06:28:03.705703 5819 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 06:28:03.705727 5819 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 06:28:03.706902 5819 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 06:28:03.706940 5819 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 06:28:03.706965 5819 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.12.129
I20260806 06:28:03.708585 5819 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.148:40857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785997581049705-21554-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785997581049705-21554-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.21.12.148:35907/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.129.keytab
--principal=kudu/127.21.12.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.12.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785997581049705-21554-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.12.129
--webserver_port=0
--tserver_master_addrs=127.21.12.190:39335
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5819
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.12.129
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785997581049705-21554-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 05:11:02 UTC on ccycloud.quasar-hvzbbz.root.comops.site
build id 4569617
I20260806 06:28:03.708781 5819 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 06:28:03.708992 5819 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 06:28:03.714322 5828 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:28:03.714403 5831 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:28:03.714570 5836 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:28:03.715303 5819 server_base.cc:1034] running on GCE node
I20260806 06:28:03.715517 5819 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 06:28:03.715734 5819 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 06:28:03.716871 5819 hybrid_clock.cc:648] HybridClock initialized: now 1785997683716866 us; error 39 us; skew 500 ppm
Aug 06 06:28:03 dist-test-slave-4tdj krb5kdc[4955](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997683, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 06:28:03.720008 5819 init.cc:377] Logged in from keytab as kudu/127.21.12.129@KRBTEST.COM (short username kudu)
I20260806 06:28:03.720429 5819 webserver.cc:460] Webserver started at http://127.21.12.129:42213/ using document root <none> and password file <none>
I20260806 06:28:03.720635 5819 fs_manager.cc:359] Metadata directory not provided
I20260806 06:28:03.720681 5819 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 06:28:03.720806 5819 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 06 06:28:03 dist-test-slave-4tdj krb5kdc[4955](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997683, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.129@KRBTEST.COM for HTTP/127.21.12.148@KRBTEST.COM
I20260806 06:28:03.736202 5819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785997581049705-21554-0/minicluster-data/ts-0/data/instance:
uuid: "be4677a62ad7407c9a534d6a8fab12b7"
format_stamp: "Formatted at 2026-08-06 06:28:03 on dist-test-slave-4tdj"
server_key: "c6eaaa96f2adaf9d0a83f02dcf969f3a"
server_key_iv: "e9e2d73a96bf1e29a954f859000de5e1"
server_key_version: "kuduclusterkey@0"
I20260806 06:28:03.736799 5819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785997581049705-21554-0/minicluster-data/ts-0/wal/instance:
uuid: "be4677a62ad7407c9a534d6a8fab12b7"
format_stamp: "Formatted at 2026-08-06 06:28:03 on dist-test-slave-4tdj"
server_key: "c6eaaa96f2adaf9d0a83f02dcf969f3a"
server_key_iv: "e9e2d73a96bf1e29a954f859000de5e1"
server_key_version: "kuduclusterkey@0"
I20260806 06:28:03.742856 5819 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.000s
I20260806 06:28:03.756565 5846 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 06:28:03.756798 5819 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 06:28:03.756912 5819 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785997581049705-21554-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785997581049705-21554-0/minicluster-data/ts-0/wal
uuid: "be4677a62ad7407c9a534d6a8fab12b7"
format_stamp: "Formatted at 2026-08-06 06:28:03 on dist-test-slave-4tdj"
server_key: "c6eaaa96f2adaf9d0a83f02dcf969f3a"
server_key_iv: "e9e2d73a96bf1e29a954f859000de5e1"
server_key_version: "kuduclusterkey@0"
I20260806 06:28:03.757010 5819 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785997581049705-21554-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785997581049705-21554-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785997581049705-21554-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 06:28:03.783649 5819 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 06:28:03.786197 5819 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 06:28:03.786478 5819 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 06:28:03.787074 5819 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 06:28:03.787563 5819 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 06:28:03.787654 5819 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 06:28:03.787756 5819 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 06:28:03.787832 5819 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 06:28:03.807242 5819 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.129:43579
I20260806 06:28:03.807365 6011 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.129:43579 every 8 connection(s)
I20260806 06:28:03.807907 5819 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785997581049705-21554-0/minicluster-data/ts-0/data/info.pb
I20260806 06:28:03.814141 21554 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHhLhT/build/release/bin/kudu as pid 5819
I20260806 06:28:03.814248 21554 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785997581049705-21554-0/minicluster-data/ts-0/wal/instance
I20260806 06:28:03.814466 21554 external_mini_cluster.cc:1428] Setting key ecc080bcd88785b720a9da07e5bcb510
Aug 06 06:28:03 dist-test-slave-4tdj krb5kdc[4955](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997683, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.129@KRBTEST.COM for kudu/127.21.12.190@KRBTEST.COM
I20260806 06:28:03.824393 6017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:28:03.808850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:39335 (local address 127.21.12.129:48061)
0806 06:28:03.809239 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0806 06:28:03.809289 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0806 06:28:03.810379 (+ 1090us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:28:03.814167 (+ 3788us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:28:03.814172 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:28:03.814668 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
0806 06:28:03.815150 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:28:03.815158 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:28:03.816064 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:28:03.816066 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:28:03.817311 (+ 1245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:28:03.817315 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:28:03.817826 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:28:03.817827 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:28:03.818005 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:28:03.818641 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 06:28:03.818652 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0806 06:28:03.820603 (+ 1951us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 06:28:03.822614 (+ 2011us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:28:03.822622 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:28:03.822625 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:28:03.822918 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:28:03.823177 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:28:03.823178 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:28:03.823178 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:28:03.823319 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:28:03.823656 (+ 337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 06:28:03.823663 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 06:28:03.823828 (+ 165us) client_negotiation.cc:769] Sending connection context
0806 06:28:03.823943 (+ 115us) client_negotiation.cc:240] Negotiation successful
0806 06:28:03.824107 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":185,"threads_started":1}
I20260806 06:28:03.825229 6015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:28:03.808874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.129:48061 (local address 127.21.12.190:39335)
0806 06:28:03.813913 (+ 5039us) server_negotiation.cc:206] Beginning negotiation
0806 06:28:03.813915 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:28:03.813925 (+ 10us) server_negotiation.cc:407] Connection header received
0806 06:28:03.813953 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:28:03.813954 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:28:03.813968 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:28:03.815253 (+ 1285us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:28:03.815300 (+ 47us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:28:03.815955 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:28:03.817421 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:28:03.817732 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:28:03.818280 (+ 548us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:28:03.820706 (+ 2426us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:28:03.820707 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:28:03.820707 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 06:28:03.820715 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:28:03.822513 (+ 1798us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:28:03.823022 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:28:03.823023 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:28:03.823024 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:28:03.823094 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:28:03.823407 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:28:03.823407 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:28:03.823408 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:28:03.823559 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:28:03.823626 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0806 06:28:03.825014 (+ 1388us) server_negotiation.cc:299] Negotiation successful
0806 06:28:03.825115 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5014,"thread_start_us":54,"threads_started":1}
I20260806 06:28:03.825883 6012 heartbeater.cc:347] Connected to a master server at 127.21.12.190:39335
I20260806 06:28:03.825991 6012 heartbeater.cc:464] Registering TS with master...
I20260806 06:28:03.826200 6012 heartbeater.cc:511] Master 127.21.12.190:39335 requested a full tablet report, sending...
I20260806 06:28:03.826802 5699 ts_manager.cc:194] Registered new tserver with Master: be4677a62ad7407c9a534d6a8fab12b7 (127.21.12.129:43579)
I20260806 06:28:03.827692 5699 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.12.129@KRBTEST.COM'} at 127.21.12.129:48061
WARNING: no policy specified for kudu/127.21.12.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.12.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.12.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.130.keytab.
Entry for principal kudu/127.21.12.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.130.keytab.
I20260806 06:28:03.860893 21554 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHhLhT/build/release/bin/kudu
/tmp/dist-test-taskFHhLhT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785997581049705-21554-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785997581049705-21554-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785997581049705-21554-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785997581049705-21554-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.21.12.148:35907/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.12.130:0
--local_ip_for_outbound_sockets=127.21.12.130
--webserver_interface=127.21.12.130
--webserver_port=0
--tserver_master_addrs=127.21.12.190:39335
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.130.keytab
--principal=kudu/127.21.12.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.148:40857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785997581049705-21554-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785997581049705-21554-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785997581049705-21554-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 06:28:04.029891 6021 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 06:28:04.030354 6021 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 06:28:04.030447 6021 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 06:28:04.032177 6021 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 06:28:04.032298 6021 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 06:28:04.032380 6021 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.12.130
I20260806 06:28:04.034732 6021 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.148:40857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785997581049705-21554-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785997581049705-21554-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.21.12.148:35907/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.130.keytab
--principal=kudu/127.21.12.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.12.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785997581049705-21554-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.12.130
--webserver_port=0
--tserver_master_addrs=127.21.12.190:39335
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6021
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.12.130
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785997581049705-21554-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 05:11:02 UTC on ccycloud.quasar-hvzbbz.root.comops.site
build id 4569617
I20260806 06:28:04.035176 6021 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 06:28:04.035480 6021 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 06:28:04.044853 6033 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:28:04.045482 6034 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:28:04.045936 6021 server_base.cc:1034] running on GCE node
W20260806 06:28:04.046260 6037 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:28:04.046588 6021 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 06:28:04.046842 6021 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 06:28:04.047988 6021 hybrid_clock.cc:648] HybridClock initialized: now 1785997684047967 us; error 34 us; skew 500 ppm
Aug 06 06:28:04 dist-test-slave-4tdj krb5kdc[4955](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997684, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 06:28:04.052048 6021 init.cc:377] Logged in from keytab as kudu/127.21.12.130@KRBTEST.COM (short username kudu)
I20260806 06:28:04.052563 6021 webserver.cc:460] Webserver started at http://127.21.12.130:45531/ using document root <none> and password file <none>
I20260806 06:28:04.052843 6021 fs_manager.cc:359] Metadata directory not provided
I20260806 06:28:04.052942 6021 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 06:28:04.053114 6021 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 06 06:28:04 dist-test-slave-4tdj krb5kdc[4955](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997684, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.130@KRBTEST.COM for HTTP/127.21.12.148@KRBTEST.COM
Aug 06 06:28:04 dist-test-slave-4tdj krb5kdc[4955](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1785997684, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 06:28:04.072319 6021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785997581049705-21554-0/minicluster-data/ts-1/data/instance:
uuid: "5bed18de14324ddebf9c7cd293d5483c"
format_stamp: "Formatted at 2026-08-06 06:28:04 on dist-test-slave-4tdj"
server_key: "bdd40e163599d0075021f006e9a62a5b"
server_key_iv: "1a3b555ad97350eb03b8fb384c071ffe"
server_key_version: "kuduclusterkey@0"
I20260806 06:28:04.072852 6021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785997581049705-21554-0/minicluster-data/ts-1/wal/instance:
uuid: "5bed18de14324ddebf9c7cd293d5483c"
format_stamp: "Formatted at 2026-08-06 06:28:04 on dist-test-slave-4tdj"
server_key: "bdd40e163599d0075021f006e9a62a5b"
server_key_iv: "1a3b555ad97350eb03b8fb384c071ffe"
server_key_version: "kuduclusterkey@0"
I20260806 06:28:04.075966 6021 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260806 06:28:04.097105 6048 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 06:28:04.097349 6021 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.001s sys 0.000s
I20260806 06:28:04.097450 6021 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785997581049705-21554-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785997581049705-21554-0/minicluster-data/ts-1/wal
uuid: "5bed18de14324ddebf9c7cd293d5483c"
format_stamp: "Formatted at 2026-08-06 06:28:04 on dist-test-slave-4tdj"
server_key: "bdd40e163599d0075021f006e9a62a5b"
server_key_iv: "1a3b555ad97350eb03b8fb384c071ffe"
server_key_version: "kuduclusterkey@0"
I20260806 06:28:04.097538 6021 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785997581049705-21554-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785997581049705-21554-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785997581049705-21554-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 06:28:04.123364 6021 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 06:28:04.131088 6021 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 06:28:04.132297 6021 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 06:28:04.132679 6021 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 06:28:04.133213 6021 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 06:28:04.133275 6021 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 06:28:04.133312 6021 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 06:28:04.133342 6021 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 06:28:04.151744 6021 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.130:36787
I20260806 06:28:04.151885 6209 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.130:36787 every 8 connection(s)
I20260806 06:28:04.152297 6021 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785997581049705-21554-0/minicluster-data/ts-1/data/info.pb
I20260806 06:28:04.154557 21554 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHhLhT/build/release/bin/kudu as pid 6021
I20260806 06:28:04.154664 21554 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785997581049705-21554-0/minicluster-data/ts-1/wal/instance
I20260806 06:28:04.154855 21554 external_mini_cluster.cc:1428] Setting key 97fe243c1fb3fa2d7a0bda2cc38c0071
Aug 06 06:28:04 dist-test-slave-4tdj krb5kdc[4955](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997684, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.130@KRBTEST.COM for kudu/127.21.12.190@KRBTEST.COM
I20260806 06:28:04.165248 6015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:28:04.153217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.130:55987 (local address 127.21.12.190:39335)
0806 06:28:04.153283 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0806 06:28:04.153285 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:28:04.153692 (+ 407us) server_negotiation.cc:407] Connection header received
0806 06:28:04.154621 (+ 929us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:28:04.154622 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:28:04.154642 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:28:04.154717 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:28:04.155811 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:28:04.156587 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:28:04.158073 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:28:04.158389 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:28:04.158463 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:28:04.161860 (+ 3397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:28:04.161862 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:28:04.161862 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 06:28:04.161870 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:28:04.163614 (+ 1744us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:28:04.164100 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:28:04.164101 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:28:04.164102 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:28:04.164165 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:28:04.164455 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:28:04.164456 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:28:04.164456 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:28:04.164590 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:28:04.164629 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0806 06:28:04.165012 (+ 383us) server_negotiation.cc:299] Negotiation successful
0806 06:28:04.165107 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260806 06:28:04.165831 6213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:28:04.153241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:39335 (local address 127.21.12.130:55987)
0806 06:28:04.153545 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0806 06:28:04.153588 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0806 06:28:04.154470 (+ 882us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:28:04.154808 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:28:04.154813 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:28:04.155259 (+ 446us) client_negotiation.cc:189] Negotiated authn=SASL
0806 06:28:04.155695 (+ 436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:28:04.155703 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:28:04.156700 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:28:04.156702 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:28:04.157968 (+ 1266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:28:04.157972 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:28:04.158533 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:28:04.158535 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:28:04.158735 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:28:04.159475 (+ 740us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 06:28:04.159486 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0806 06:28:04.161736 (+ 2250us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 06:28:04.163713 (+ 1977us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:28:04.163720 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:28:04.163722 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:28:04.164004 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:28:04.164251 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:28:04.164252 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:28:04.164252 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:28:04.164383 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:28:04.164678 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 06:28:04.164684 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 06:28:04.164844 (+ 160us) client_negotiation.cc:769] Sending connection context
0806 06:28:04.165386 (+ 542us) client_negotiation.cc:240] Negotiation successful
0806 06:28:04.165549 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":174,"threads_started":1}
I20260806 06:28:04.166518 6211 heartbeater.cc:347] Connected to a master server at 127.21.12.190:39335
I20260806 06:28:04.166632 6211 heartbeater.cc:464] Registering TS with master...
I20260806 06:28:04.166838 6211 heartbeater.cc:511] Master 127.21.12.190:39335 requested a full tablet report, sending...
I20260806 06:28:04.167362 5699 ts_manager.cc:194] Registered new tserver with Master: 5bed18de14324ddebf9c7cd293d5483c (127.21.12.130:36787)
I20260806 06:28:04.167858 5699 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.12.130@KRBTEST.COM'} at 127.21.12.130:55987
WARNING: no policy specified for kudu/127.21.12.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.12.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.12.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.131.keytab.
Entry for principal kudu/127.21.12.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.131.keytab.
I20260806 06:28:04.190536 21554 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHhLhT/build/release/bin/kudu
/tmp/dist-test-taskFHhLhT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785997581049705-21554-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785997581049705-21554-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785997581049705-21554-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785997581049705-21554-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.21.12.148:35907/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.12.131:0
--local_ip_for_outbound_sockets=127.21.12.131
--webserver_interface=127.21.12.131
--webserver_port=0
--tserver_master_addrs=127.21.12.190:39335
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.131.keytab
--principal=kudu/127.21.12.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.148:40857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785997581049705-21554-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785997581049705-21554-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785997581049705-21554-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 06:28:04.336527 6219 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 06:28:04.336867 6219 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 06:28:04.336952 6219 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 06:28:04.338801 6219 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 06:28:04.338860 6219 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 06:28:04.338899 6219 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.12.131
I20260806 06:28:04.341162 6219 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.148:40857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785997581049705-21554-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785997581049705-21554-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.21.12.148:35907/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.131.keytab
--principal=kudu/127.21.12.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.12.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785997581049705-21554-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.12.131
--webserver_port=0
--tserver_master_addrs=127.21.12.190:39335
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6219
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.12.131
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785997581049705-21554-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 05:11:02 UTC on ccycloud.quasar-hvzbbz.root.comops.site
build id 4569617
I20260806 06:28:04.341408 6219 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 06:28:04.341701 6219 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 06:28:04.350852 6231 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:28:04.352366 6219 server_base.cc:1034] running on GCE node
W20260806 06:28:04.352639 6233 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:28:04.353137 6230 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:28:04.353482 6219 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 06:28:04.353879 6219 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 06:28:04.355064 6219 hybrid_clock.cc:648] HybridClock initialized: now 1785997684355044 us; error 37 us; skew 500 ppm
Aug 06 06:28:04 dist-test-slave-4tdj krb5kdc[4955](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997684, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 06:28:04.358904 6219 init.cc:377] Logged in from keytab as kudu/127.21.12.131@KRBTEST.COM (short username kudu)
I20260806 06:28:04.359431 6219 webserver.cc:460] Webserver started at http://127.21.12.131:33499/ using document root <none> and password file <none>
I20260806 06:28:04.359678 6219 fs_manager.cc:359] Metadata directory not provided
I20260806 06:28:04.359728 6219 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 06:28:04.359849 6219 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 06 06:28:04 dist-test-slave-4tdj krb5kdc[4955](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997684, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.131@KRBTEST.COM for HTTP/127.21.12.148@KRBTEST.COM
I20260806 06:28:04.373891 6219 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785997581049705-21554-0/minicluster-data/ts-2/data/instance:
uuid: "9c3bc372e32e4b9c8c4a80387367df84"
format_stamp: "Formatted at 2026-08-06 06:28:04 on dist-test-slave-4tdj"
server_key: "8ec7ca90e64e05530b4f4ff22816a817"
server_key_iv: "8bb3afd46155ae576bfa5ba4e2361b53"
server_key_version: "kuduclusterkey@0"
I20260806 06:28:04.374485 6219 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785997581049705-21554-0/minicluster-data/ts-2/wal/instance:
uuid: "9c3bc372e32e4b9c8c4a80387367df84"
format_stamp: "Formatted at 2026-08-06 06:28:04 on dist-test-slave-4tdj"
server_key: "8ec7ca90e64e05530b4f4ff22816a817"
server_key_iv: "8bb3afd46155ae576bfa5ba4e2361b53"
server_key_version: "kuduclusterkey@0"
I20260806 06:28:04.377704 6219 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.001s
I20260806 06:28:04.394559 6242 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 06:28:04.394856 6219 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 06:28:04.395008 6219 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785997581049705-21554-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785997581049705-21554-0/minicluster-data/ts-2/wal
uuid: "9c3bc372e32e4b9c8c4a80387367df84"
format_stamp: "Formatted at 2026-08-06 06:28:04 on dist-test-slave-4tdj"
server_key: "8ec7ca90e64e05530b4f4ff22816a817"
server_key_iv: "8bb3afd46155ae576bfa5ba4e2361b53"
server_key_version: "kuduclusterkey@0"
I20260806 06:28:04.395102 6219 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785997581049705-21554-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785997581049705-21554-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785997581049705-21554-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 06:28:04.405318 6219 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 06:28:04.407320 6219 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 06:28:04.407429 6219 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 06:28:04.407763 6219 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 06:28:04.408123 6219 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 06:28:04.408161 6219 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 06:28:04.408190 6219 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 06:28:04.408216 6219 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 06:28:04.422122 6219 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.131:37629
I20260806 06:28:04.422364 6414 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.131:37629 every 8 connection(s)
I20260806 06:28:04.422585 6219 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785997581049705-21554-0/minicluster-data/ts-2/data/info.pb
I20260806 06:28:04.428925 21554 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHhLhT/build/release/bin/kudu as pid 6219
I20260806 06:28:04.429013 21554 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785997581049705-21554-0/minicluster-data/ts-2/wal/instance
I20260806 06:28:04.429167 21554 external_mini_cluster.cc:1428] Setting key a4ede0bacc642f79216565d8023c823d
Aug 06 06:28:04 dist-test-slave-4tdj krb5kdc[4955](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997684, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.131@KRBTEST.COM for kudu/127.21.12.190@KRBTEST.COM
I20260806 06:28:04.433856 6015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:28:04.423439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.131:49861 (local address 127.21.12.190:39335)
0806 06:28:04.423508 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0806 06:28:04.423511 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 06:28:04.423970 (+ 459us) server_negotiation.cc:407] Connection header received
0806 06:28:04.424992 (+ 1022us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:28:04.424993 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:28:04.425003 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:28:04.425051 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:28:04.426097 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:28:04.426728 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:28:04.428237 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:28:04.428427 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:28:04.428481 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:28:04.431261 (+ 2780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:28:04.431263 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:28:04.431263 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 06:28:04.431272 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:28:04.432502 (+ 1230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:28:04.432959 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:28:04.432960 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:28:04.432960 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:28:04.433010 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:28:04.433262 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:28:04.433262 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:28:04.433263 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:28:04.433368 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:28:04.433411 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0806 06:28:04.433730 (+ 319us) server_negotiation.cc:299] Negotiation successful
0806 06:28:04.433794 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260806 06:28:04.434135 6418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:28:04.423408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:39335 (local address 127.21.12.131:49861)
0806 06:28:04.423796 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0806 06:28:04.423846 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0806 06:28:04.424878 (+ 1032us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:28:04.425135 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:28:04.425140 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:28:04.425593 (+ 453us) client_negotiation.cc:189] Negotiated authn=SASL
0806 06:28:04.426013 (+ 420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:28:04.426020 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:28:04.426816 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:28:04.426818 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:28:04.428149 (+ 1331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:28:04.428152 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:28:04.428512 (+ 360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:28:04.428513 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:28:04.428686 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:28:04.429334 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 06:28:04.429343 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0806 06:28:04.431156 (+ 1813us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 06:28:04.432580 (+ 1424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:28:04.432587 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:28:04.432589 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:28:04.432893 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:28:04.433064 (+ 171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:28:04.433064 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:28:04.433065 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:28:04.433204 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:28:04.433422 (+ 218us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 06:28:04.433428 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 06:28:04.433596 (+ 168us) client_negotiation.cc:769] Sending connection context
0806 06:28:04.433716 (+ 120us) client_negotiation.cc:240] Negotiation successful
0806 06:28:04.433871 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":195,"threads_started":1}
I20260806 06:28:04.434641 6415 heartbeater.cc:347] Connected to a master server at 127.21.12.190:39335
I20260806 06:28:04.434737 6415 heartbeater.cc:464] Registering TS with master...
I20260806 06:28:04.434900 6415 heartbeater.cc:511] Master 127.21.12.190:39335 requested a full tablet report, sending...
I20260806 06:28:04.435253 5699 ts_manager.cc:194] Registered new tserver with Master: 9c3bc372e32e4b9c8c4a80387367df84 (127.21.12.131:37629)
I20260806 06:28:04.435611 5699 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.12.131@KRBTEST.COM'} at 127.21.12.131:49861
I20260806 06:28:04.440699 21554 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 06:28:04.448189 6015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:28:04.442169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54582 (local address 127.21.12.190:39335)
0806 06:28:04.442208 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0806 06:28:04.442209 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0806 06:28:04.442464 (+ 255us) server_negotiation.cc:407] Connection header received
0806 06:28:04.442544 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:28:04.442544 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:28:04.442555 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:28:04.442599 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:28:04.443194 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:28:04.443677 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:28:04.444432 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:28:04.444631 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:28:04.444683 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:28:04.445938 (+ 1255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:28:04.445940 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:28:04.445940 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 06:28:04.445947 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:28:04.447186 (+ 1239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:28:04.447499 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:28:04.447500 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:28:04.447501 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:28:04.447548 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:28:04.447711 (+ 163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:28:04.447712 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:28:04.447712 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:28:04.447818 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:28:04.447906 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0806 06:28:04.448031 (+ 125us) server_negotiation.cc:299] Negotiation successful
0806 06:28:04.448103 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260806 06:28:04.449652 5699 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54582:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 06:28:04.451411 5699 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 06:28:04.459780 6438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:28:04.454334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.129:43579 (local address 127.0.0.1:32922)
0806 06:28:04.455461 (+ 1127us) negotiation.cc:107] Waiting for socket to connect
0806 06:28:04.455506 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0806 06:28:04.455576 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:28:04.456120 (+ 544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:28:04.456121 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:28:04.456122 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 06:28:04.456152 (+ 30us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:28:04.456153 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:28:04.457566 (+ 1413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:28:04.457567 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:28:04.458577 (+ 1010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:28:04.458579 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:28:04.459384 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:28:04.459385 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:28:04.459532 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:28:04.459581 (+ 49us) client_negotiation.cc:769] Sending connection context
0806 06:28:04.459644 (+ 63us) client_negotiation.cc:240] Negotiation successful
0806 06:28:04.459696 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1072,"thread_start_us":136,"threads_started":1}
I20260806 06:28:04.459959 6434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:28:04.454519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56678 (local address 127.21.12.131:37629)
0806 06:28:04.455348 (+ 829us) server_negotiation.cc:206] Beginning negotiation
0806 06:28:04.455353 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 06:28:04.455821 (+ 468us) server_negotiation.cc:407] Connection header received
0806 06:28:04.455901 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:28:04.455903 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:28:04.455917 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:28:04.455988 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 06:28:04.456208 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:28:04.457060 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:28:04.458580 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:28:04.459520 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:28:04.459606 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:28:04.459759 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0806 06:28:04.459791 (+ 32us) server_negotiation.cc:299] Negotiation successful
0806 06:28:04.459833 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":765,"thread_start_us":139,"threads_started":1}
I20260806 06:28:04.459789 6433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:28:04.454477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.131:37629 (local address 127.0.0.1:56678)
0806 06:28:04.455748 (+ 1271us) negotiation.cc:107] Waiting for socket to connect
0806 06:28:04.455759 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0806 06:28:04.455818 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:28:04.456048 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:28:04.456052 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:28:04.456054 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 06:28:04.456115 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:28:04.456117 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:28:04.457145 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:28:04.457146 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:28:04.458362 (+ 1216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:28:04.458364 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:28:04.459605 (+ 1241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:28:04.459606 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:28:04.459661 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:28:04.459663 (+ 2us) client_negotiation.cc:769] Sending connection context
0806 06:28:04.459699 (+ 36us) client_negotiation.cc:240] Negotiation successful
0806 06:28:04.459730 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1239,"thread_start_us":99,"threads_started":1}
I20260806 06:28:04.460325 6436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:28:04.454661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32922 (local address 127.21.12.129:43579)
0806 06:28:04.455936 (+ 1275us) server_negotiation.cc:206] Beginning negotiation
0806 06:28:04.455941 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 06:28:04.455951 (+ 10us) server_negotiation.cc:407] Connection header received
0806 06:28:04.455987 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:28:04.455989 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:28:04.456006 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:28:04.456121 (+ 115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 06:28:04.456200 (+ 79us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:28:04.457444 (+ 1244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:28:04.458646 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:28:04.459201 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:28:04.459267 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:28:04.459333 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0806 06:28:04.460209 (+ 876us) server_negotiation.cc:299] Negotiation successful
0806 06:28:04.460259 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1237,"thread_start_us":50,"threads_started":1}
I20260806 06:28:04.460912 6309 tablet_service.cc:1467] Processing CreateTablet for tablet afde7296be6e4c00b71b1e30dd0c1bcd (DEFAULT_TABLE table=test-table [id=72c88d1fe591467e94f5415a853bfd69]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 06:28:04.461016 5908 tablet_service.cc:1467] Processing CreateTablet for tablet afde7296be6e4c00b71b1e30dd0c1bcd (DEFAULT_TABLE table=test-table [id=72c88d1fe591467e94f5415a853bfd69]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 06:28:04.461391 6309 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet afde7296be6e4c00b71b1e30dd0c1bcd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 06:28:04.461421 5908 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet afde7296be6e4c00b71b1e30dd0c1bcd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 06:28:04.464900 6438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:28:04.454981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.130:36787 (local address 127.0.0.1:54574)
0806 06:28:04.460048 (+ 5067us) negotiation.cc:107] Waiting for socket to connect
0806 06:28:04.460058 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0806 06:28:04.460094 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:28:04.460647 (+ 553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:28:04.460648 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:28:04.460650 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 06:28:04.460682 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:28:04.460683 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:28:04.462553 (+ 1870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:28:04.462554 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:28:04.463575 (+ 1021us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:28:04.463578 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:28:04.464706 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:28:04.464707 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:28:04.464763 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:28:04.464767 (+ 4us) client_negotiation.cc:769] Sending connection context
0806 06:28:04.464794 (+ 27us) client_negotiation.cc:240] Negotiation successful
0806 06:28:04.464836 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5055,"thread_start_us":124,"threads_started":1}
I20260806 06:28:04.465437 6437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:28:04.454757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54574 (local address 127.21.12.130:36787)
0806 06:28:04.455079 (+ 322us) server_negotiation.cc:206] Beginning negotiation
0806 06:28:04.455084 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 06:28:04.460414 (+ 5330us) server_negotiation.cc:407] Connection header received
0806 06:28:04.460507 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:28:04.460510 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:28:04.460539 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:28:04.460745 (+ 206us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 06:28:04.460880 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:28:04.462447 (+ 1567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:28:04.463772 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:28:04.464608 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:28:04.465077 (+ 469us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:28:04.465246 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0806 06:28:04.465275 (+ 29us) server_negotiation.cc:299] Negotiation successful
0806 06:28:04.465316 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":257,"thread_start_us":140,"threads_started":1}
I20260806 06:28:04.466495 6109 tablet_service.cc:1467] Processing CreateTablet for tablet afde7296be6e4c00b71b1e30dd0c1bcd (DEFAULT_TABLE table=test-table [id=72c88d1fe591467e94f5415a853bfd69]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 06:28:04.466828 6440 tablet_bootstrap.cc:492] T afde7296be6e4c00b71b1e30dd0c1bcd P 9c3bc372e32e4b9c8c4a80387367df84: Bootstrap starting.
I20260806 06:28:04.466943 6109 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet afde7296be6e4c00b71b1e30dd0c1bcd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 06:28:04.468274 6440 tablet_bootstrap.cc:654] T afde7296be6e4c00b71b1e30dd0c1bcd P 9c3bc372e32e4b9c8c4a80387367df84: Neither blocks nor log segments found. Creating new log.
I20260806 06:28:04.469058 6440 log.cc:824] T afde7296be6e4c00b71b1e30dd0c1bcd P 9c3bc372e32e4b9c8c4a80387367df84: Log is configured to *not* fsync() on all Append() calls
I20260806 06:28:04.470001 6440 tablet_bootstrap.cc:492] T afde7296be6e4c00b71b1e30dd0c1bcd P 9c3bc372e32e4b9c8c4a80387367df84: No bootstrap required, opened a new log
I20260806 06:28:04.470077 6440 ts_tablet_manager.cc:1397] T afde7296be6e4c00b71b1e30dd0c1bcd P 9c3bc372e32e4b9c8c4a80387367df84: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20260806 06:28:04.471284 6444 tablet_bootstrap.cc:492] T afde7296be6e4c00b71b1e30dd0c1bcd P 5bed18de14324ddebf9c7cd293d5483c: Bootstrap starting.
I20260806 06:28:04.473265 6444 tablet_bootstrap.cc:654] T afde7296be6e4c00b71b1e30dd0c1bcd P 5bed18de14324ddebf9c7cd293d5483c: Neither blocks nor log segments found. Creating new log.
I20260806 06:28:04.473826 6440 raft_consensus.cc:348] T afde7296be6e4c00b71b1e30dd0c1bcd P 9c3bc372e32e4b9c8c4a80387367df84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be4677a62ad7407c9a534d6a8fab12b7" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 43579 } } peers { permanent_uuid: "5bed18de14324ddebf9c7cd293d5483c" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 36787 } } peers { permanent_uuid: "9c3bc372e32e4b9c8c4a80387367df84" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 37629 } }
I20260806 06:28:04.474009 6440 raft_consensus.cc:374] T afde7296be6e4c00b71b1e30dd0c1bcd P 9c3bc372e32e4b9c8c4a80387367df84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 06:28:04.474087 6440 raft_consensus.cc:729] T afde7296be6e4c00b71b1e30dd0c1bcd P 9c3bc372e32e4b9c8c4a80387367df84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c3bc372e32e4b9c8c4a80387367df84, State: Initialized, Role: FOLLOWER
I20260806 06:28:04.474093 6444 log.cc:824] T afde7296be6e4c00b71b1e30dd0c1bcd P 5bed18de14324ddebf9c7cd293d5483c: Log is configured to *not* fsync() on all Append() calls
I20260806 06:28:04.474242 6440 consensus_queue.cc:260] T afde7296be6e4c00b71b1e30dd0c1bcd P 9c3bc372e32e4b9c8c4a80387367df84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be4677a62ad7407c9a534d6a8fab12b7" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 43579 } } peers { permanent_uuid: "5bed18de14324ddebf9c7cd293d5483c" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 36787 } } peers { permanent_uuid: "9c3bc372e32e4b9c8c4a80387367df84" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 37629 } }
I20260806 06:28:04.474563 6440 ts_tablet_manager.cc:1428] T afde7296be6e4c00b71b1e30dd0c1bcd P 9c3bc372e32e4b9c8c4a80387367df84: Time spent starting tablet: real 0.004s user 0.001s sys 0.004s
I20260806 06:28:04.474678 6415 heartbeater.cc:503] Master 127.21.12.190:39335 was elected leader, sending a full tablet report...
I20260806 06:28:04.475230 6444 tablet_bootstrap.cc:492] T afde7296be6e4c00b71b1e30dd0c1bcd P 5bed18de14324ddebf9c7cd293d5483c: No bootstrap required, opened a new log
I20260806 06:28:04.475293 6444 ts_tablet_manager.cc:1397] T afde7296be6e4c00b71b1e30dd0c1bcd P 5bed18de14324ddebf9c7cd293d5483c: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260806 06:28:04.478196 6444 raft_consensus.cc:348] T afde7296be6e4c00b71b1e30dd0c1bcd P 5bed18de14324ddebf9c7cd293d5483c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be4677a62ad7407c9a534d6a8fab12b7" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 43579 } } peers { permanent_uuid: "5bed18de14324ddebf9c7cd293d5483c" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 36787 } } peers { permanent_uuid: "9c3bc372e32e4b9c8c4a80387367df84" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 37629 } }
I20260806 06:28:04.478353 6444 raft_consensus.cc:374] T afde7296be6e4c00b71b1e30dd0c1bcd P 5bed18de14324ddebf9c7cd293d5483c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 06:28:04.478422 6444 raft_consensus.cc:729] T afde7296be6e4c00b71b1e30dd0c1bcd P 5bed18de14324ddebf9c7cd293d5483c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5bed18de14324ddebf9c7cd293d5483c, State: Initialized, Role: FOLLOWER
I20260806 06:28:04.478464 6439 tablet_bootstrap.cc:492] T afde7296be6e4c00b71b1e30dd0c1bcd P be4677a62ad7407c9a534d6a8fab12b7: Bootstrap starting.
I20260806 06:28:04.478541 6444 consensus_queue.cc:260] T afde7296be6e4c00b71b1e30dd0c1bcd P 5bed18de14324ddebf9c7cd293d5483c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be4677a62ad7407c9a534d6a8fab12b7" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 43579 } } peers { permanent_uuid: "5bed18de14324ddebf9c7cd293d5483c" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 36787 } } peers { permanent_uuid: "9c3bc372e32e4b9c8c4a80387367df84" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 37629 } }
I20260806 06:28:04.478873 6444 ts_tablet_manager.cc:1428] T afde7296be6e4c00b71b1e30dd0c1bcd P 5bed18de14324ddebf9c7cd293d5483c: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260806 06:28:04.479051 6211 heartbeater.cc:503] Master 127.21.12.190:39335 was elected leader, sending a full tablet report...
I20260806 06:28:04.480597 6439 tablet_bootstrap.cc:654] T afde7296be6e4c00b71b1e30dd0c1bcd P be4677a62ad7407c9a534d6a8fab12b7: Neither blocks nor log segments found. Creating new log.
I20260806 06:28:04.481488 6439 log.cc:824] T afde7296be6e4c00b71b1e30dd0c1bcd P be4677a62ad7407c9a534d6a8fab12b7: Log is configured to *not* fsync() on all Append() calls
I20260806 06:28:04.482607 6439 tablet_bootstrap.cc:492] T afde7296be6e4c00b71b1e30dd0c1bcd P be4677a62ad7407c9a534d6a8fab12b7: No bootstrap required, opened a new log
I20260806 06:28:04.482702 6439 ts_tablet_manager.cc:1397] T afde7296be6e4c00b71b1e30dd0c1bcd P be4677a62ad7407c9a534d6a8fab12b7: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260806 06:28:04.487530 6439 raft_consensus.cc:348] T afde7296be6e4c00b71b1e30dd0c1bcd P be4677a62ad7407c9a534d6a8fab12b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be4677a62ad7407c9a534d6a8fab12b7" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 43579 } } peers { permanent_uuid: "5bed18de14324ddebf9c7cd293d5483c" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 36787 } } peers { permanent_uuid: "9c3bc372e32e4b9c8c4a80387367df84" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 37629 } }
I20260806 06:28:04.487715 6439 raft_consensus.cc:374] T afde7296be6e4c00b71b1e30dd0c1bcd P be4677a62ad7407c9a534d6a8fab12b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 06:28:04.487764 6439 raft_consensus.cc:729] T afde7296be6e4c00b71b1e30dd0c1bcd P be4677a62ad7407c9a534d6a8fab12b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be4677a62ad7407c9a534d6a8fab12b7, State: Initialized, Role: FOLLOWER
I20260806 06:28:04.487890 6439 consensus_queue.cc:260] T afde7296be6e4c00b71b1e30dd0c1bcd P be4677a62ad7407c9a534d6a8fab12b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be4677a62ad7407c9a534d6a8fab12b7" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 43579 } } peers { permanent_uuid: "5bed18de14324ddebf9c7cd293d5483c" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 36787 } } peers { permanent_uuid: "9c3bc372e32e4b9c8c4a80387367df84" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 37629 } }
I20260806 06:28:04.488214 6439 ts_tablet_manager.cc:1428] T afde7296be6e4c00b71b1e30dd0c1bcd P be4677a62ad7407c9a534d6a8fab12b7: Time spent starting tablet: real 0.005s user 0.000s sys 0.006s
I20260806 06:28:04.488292 6012 heartbeater.cc:503] Master 127.21.12.190:39335 was elected leader, sending a full tablet report...
I20260806 06:28:04.489100 6450 raft_consensus.cc:482] T afde7296be6e4c00b71b1e30dd0c1bcd P be4677a62ad7407c9a534d6a8fab12b7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 06:28:04.489256 6450 raft_consensus.cc:504] T afde7296be6e4c00b71b1e30dd0c1bcd P be4677a62ad7407c9a534d6a8fab12b7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be4677a62ad7407c9a534d6a8fab12b7" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 43579 } } peers { permanent_uuid: "5bed18de14324ddebf9c7cd293d5483c" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 36787 } } peers { permanent_uuid: "9c3bc372e32e4b9c8c4a80387367df84" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 37629 } }
I20260806 06:28:04.489693 6450 leader_election.cc:288] T afde7296be6e4c00b71b1e30dd0c1bcd P be4677a62ad7407c9a534d6a8fab12b7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5bed18de14324ddebf9c7cd293d5483c (127.21.12.130:36787), 9c3bc372e32e4b9c8c4a80387367df84 (127.21.12.131:37629)
I20260806 06:28:04.494388 6434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:28:04.490368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.129:47627 (local address 127.21.12.131:37629)
0806 06:28:04.490406 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0806 06:28:04.490408 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:28:04.490415 (+ 7us) server_negotiation.cc:407] Connection header received
0806 06:28:04.490437 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:28:04.490439 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:28:04.490450 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:28:04.490502 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 06:28:04.490886 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:28:04.491670 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:28:04.493340 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:28:04.493941 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:28:04.493984 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:28:04.493992 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0806 06:28:04.494262 (+ 270us) server_negotiation.cc:299] Negotiation successful
0806 06:28:04.494326 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260806 06:28:04.494635 6451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:28:04.489803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.131:37629 (local address 127.21.12.129:47627)
0806 06:28:04.490112 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0806 06:28:04.490124 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 06:28:04.490179 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:28:04.490553 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:28:04.490555 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:28:04.490557 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 06:28:04.490797 (+ 240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:28:04.490800 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:28:04.491850 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:28:04.491851 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:28:04.493243 (+ 1392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:28:04.493246 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:28:04.494048 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:28:04.494049 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:28:04.494176 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:28:04.494181 (+ 5us) client_negotiation.cc:769] Sending connection context
0806 06:28:04.494463 (+ 282us) client_negotiation.cc:240] Negotiation successful
0806 06:28:04.494507 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":141,"threads_started":1}
I20260806 06:28:04.497974 6451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:28:04.489805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.130:36787 (local address 127.21.12.129:38063)
0806 06:28:04.494770 (+ 4965us) negotiation.cc:107] Waiting for socket to connect
0806 06:28:04.494781 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0806 06:28:04.494847 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:28:04.494998 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:28:04.494999 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:28:04.495000 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 06:28:04.495026 (+ 26us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:28:04.495027 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:28:04.496027 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:28:04.496028 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:28:04.497032 (+ 1004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:28:04.497034 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:28:04.497753 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:28:04.497753 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:28:04.497808 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:28:04.497811 (+ 3us) client_negotiation.cc:769] Sending connection context
0806 06:28:04.497876 (+ 65us) client_negotiation.cc:240] Negotiation successful
0806 06:28:04.497907 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4953,"thread_start_us":143,"threads_started":1}
I20260806 06:28:04.498013 6437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:28:04.490030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.129:38063 (local address 127.21.12.130:36787)
0806 06:28:04.490074 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0806 06:28:04.490076 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:28:04.494823 (+ 4747us) server_negotiation.cc:407] Connection header received
0806 06:28:04.494908 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:28:04.494909 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:28:04.494920 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:28:04.494946 (+ 26us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 06:28:04.495087 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:28:04.495807 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:28:04.497121 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:28:04.497640 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:28:04.497680 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:28:04.497687 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0806 06:28:04.497900 (+ 213us) server_negotiation.cc:299] Negotiation successful
0806 06:28:04.497949 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260806 06:28:04.498387 6342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afde7296be6e4c00b71b1e30dd0c1bcd" candidate_uuid: "be4677a62ad7407c9a534d6a8fab12b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c3bc372e32e4b9c8c4a80387367df84" is_pre_election: true
I20260806 06:28:04.498622 6342 raft_consensus.cc:2454] T afde7296be6e4c00b71b1e30dd0c1bcd P 9c3bc372e32e4b9c8c4a80387367df84 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be4677a62ad7407c9a534d6a8fab12b7 in term 0.
I20260806 06:28:04.498911 6147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afde7296be6e4c00b71b1e30dd0c1bcd" candidate_uuid: "be4677a62ad7407c9a534d6a8fab12b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5bed18de14324ddebf9c7cd293d5483c" is_pre_election: true
I20260806 06:28:04.499385 6147 raft_consensus.cc:2454] T afde7296be6e4c00b71b1e30dd0c1bcd P 5bed18de14324ddebf9c7cd293d5483c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be4677a62ad7407c9a534d6a8fab12b7 in term 0.
I20260806 06:28:04.499416 5849 leader_election.cc:302] T afde7296be6e4c00b71b1e30dd0c1bcd P be4677a62ad7407c9a534d6a8fab12b7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c3bc372e32e4b9c8c4a80387367df84, be4677a62ad7407c9a534d6a8fab12b7; no voters:
I20260806 06:28:04.499548 6450 raft_consensus.cc:2781] T afde7296be6e4c00b71b1e30dd0c1bcd P be4677a62ad7407c9a534d6a8fab12b7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 06:28:04.499595 6450 raft_consensus.cc:482] T afde7296be6e4c00b71b1e30dd0c1bcd P be4677a62ad7407c9a534d6a8fab12b7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 06:28:04.499615 6450 raft_consensus.cc:3037] T afde7296be6e4c00b71b1e30dd0c1bcd P be4677a62ad7407c9a534d6a8fab12b7 [term 0 FOLLOWER]: Advancing to term 1
I20260806 06:28:04.500901 6450 raft_consensus.cc:504] T afde7296be6e4c00b71b1e30dd0c1bcd P be4677a62ad7407c9a534d6a8fab12b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be4677a62ad7407c9a534d6a8fab12b7" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 43579 } } peers { permanent_uuid: "5bed18de14324ddebf9c7cd293d5483c" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 36787 } } peers { permanent_uuid: "9c3bc372e32e4b9c8c4a80387367df84" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 37629 } }
I20260806 06:28:04.501036 6450 leader_election.cc:288] T afde7296be6e4c00b71b1e30dd0c1bcd P be4677a62ad7407c9a534d6a8fab12b7 [CANDIDATE]: Term 1 election: Requested vote from peers 5bed18de14324ddebf9c7cd293d5483c (127.21.12.130:36787), 9c3bc372e32e4b9c8c4a80387367df84 (127.21.12.131:37629)
I20260806 06:28:04.501173 6147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afde7296be6e4c00b71b1e30dd0c1bcd" candidate_uuid: "be4677a62ad7407c9a534d6a8fab12b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5bed18de14324ddebf9c7cd293d5483c"
I20260806 06:28:04.501251 6147 raft_consensus.cc:3037] T afde7296be6e4c00b71b1e30dd0c1bcd P 5bed18de14324ddebf9c7cd293d5483c [term 0 FOLLOWER]: Advancing to term 1
I20260806 06:28:04.501302 6342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afde7296be6e4c00b71b1e30dd0c1bcd" candidate_uuid: "be4677a62ad7407c9a534d6a8fab12b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c3bc372e32e4b9c8c4a80387367df84"
I20260806 06:28:04.501368 6342 raft_consensus.cc:3037] T afde7296be6e4c00b71b1e30dd0c1bcd P 9c3bc372e32e4b9c8c4a80387367df84 [term 0 FOLLOWER]: Advancing to term 1
I20260806 06:28:04.502817 6147 raft_consensus.cc:2454] T afde7296be6e4c00b71b1e30dd0c1bcd P 5bed18de14324ddebf9c7cd293d5483c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be4677a62ad7407c9a534d6a8fab12b7 in term 1.
I20260806 06:28:04.502858 6342 raft_consensus.cc:2454] T afde7296be6e4c00b71b1e30dd0c1bcd P 9c3bc372e32e4b9c8c4a80387367df84 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be4677a62ad7407c9a534d6a8fab12b7 in term 1.
I20260806 06:28:04.503012 5849 leader_election.cc:302] T afde7296be6e4c00b71b1e30dd0c1bcd P be4677a62ad7407c9a534d6a8fab12b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c3bc372e32e4b9c8c4a80387367df84, be4677a62ad7407c9a534d6a8fab12b7; no voters:
I20260806 06:28:04.503114 6450 raft_consensus.cc:2781] T afde7296be6e4c00b71b1e30dd0c1bcd P be4677a62ad7407c9a534d6a8fab12b7 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 06:28:04.503289 6450 raft_consensus.cc:686] T afde7296be6e4c00b71b1e30dd0c1bcd P be4677a62ad7407c9a534d6a8fab12b7 [term 1 LEADER]: Becoming Leader. State: Replica: be4677a62ad7407c9a534d6a8fab12b7, State: Running, Role: LEADER
I20260806 06:28:04.503415 6450 consensus_queue.cc:237] T afde7296be6e4c00b71b1e30dd0c1bcd P be4677a62ad7407c9a534d6a8fab12b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be4677a62ad7407c9a534d6a8fab12b7" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 43579 } } peers { permanent_uuid: "5bed18de14324ddebf9c7cd293d5483c" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 36787 } } peers { permanent_uuid: "9c3bc372e32e4b9c8c4a80387367df84" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 37629 } }
I20260806 06:28:04.504237 5699 catalog_manager.cc:5495] T afde7296be6e4c00b71b1e30dd0c1bcd P be4677a62ad7407c9a534d6a8fab12b7 reported cstate change: term changed from 0 to 1, leader changed from <none> to be4677a62ad7407c9a534d6a8fab12b7 (127.21.12.129). New cstate: current_term: 1 leader_uuid: "be4677a62ad7407c9a534d6a8fab12b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be4677a62ad7407c9a534d6a8fab12b7" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 43579 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5bed18de14324ddebf9c7cd293d5483c" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 36787 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c3bc372e32e4b9c8c4a80387367df84" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 37629 } health_report { overall_health: UNKNOWN } } }
I20260806 06:28:04.516405 6436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:28:04.513054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32930 (local address 127.21.12.129:43579)
0806 06:28:04.513091 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0806 06:28:04.513093 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:28:04.513188 (+ 95us) server_negotiation.cc:407] Connection header received
0806 06:28:04.513236 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:28:04.513237 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:28:04.513256 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:28:04.513317 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 06:28:04.513438 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:28:04.514135 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:28:04.515270 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:28:04.515575 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:28:04.515637 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:28:04.515880 (+ 243us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 06:28:04.516001 (+ 121us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 06:28:04.516086 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0806 06:28:04.516315 (+ 229us) server_negotiation.cc:299] Negotiation successful
0806 06:28:04.516338 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260806 06:28:04.518684 6147 raft_consensus.cc:1261] T afde7296be6e4c00b71b1e30dd0c1bcd P 5bed18de14324ddebf9c7cd293d5483c [term 1 FOLLOWER]: Refusing update from remote peer be4677a62ad7407c9a534d6a8fab12b7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 06:28:04.518694 6342 raft_consensus.cc:1261] T afde7296be6e4c00b71b1e30dd0c1bcd P 9c3bc372e32e4b9c8c4a80387367df84 [term 1 FOLLOWER]: Refusing update from remote peer be4677a62ad7407c9a534d6a8fab12b7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 06:28:04.519052 6453 consensus_queue.cc:1035] T afde7296be6e4c00b71b1e30dd0c1bcd P be4677a62ad7407c9a534d6a8fab12b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c3bc372e32e4b9c8c4a80387367df84" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 37629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 06:28:04.519282 6450 consensus_queue.cc:1035] T afde7296be6e4c00b71b1e30dd0c1bcd P be4677a62ad7407c9a534d6a8fab12b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5bed18de14324ddebf9c7cd293d5483c" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 36787 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 06:28:04.523524 6456 mvcc.cc:205] Tried to move back new op lower bound from 7315446515786051584 to 7315446515726700544. Current Snapshot: MvccSnapshot[applied={T|T < 7315446515786051584}]
I20260806 06:28:04.525162 6458 mvcc.cc:205] Tried to move back new op lower bound from 7315446515786051584 to 7315446515726700544. Current Snapshot: MvccSnapshot[applied={T|T < 7315446515786051584}]
I20260806 06:28:04.528069 5699 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54582:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 06:28:04.528162 5699 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54582:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 06:28:04.528944 5699 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 62571dafbdd1430ab9a29c75964ce261: Sending DeleteTablet for 3 replicas of tablet afde7296be6e4c00b71b1e30dd0c1bcd
I20260806 06:28:04.529408 6309 tablet_service.cc:1514] Processing DeleteTablet for tablet afde7296be6e4c00b71b1e30dd0c1bcd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 06:28:04 UTC) from {username='kudu', principal='kudu/127.21.12.190@KRBTEST.COM'} at 127.0.0.1:56678
I20260806 06:28:04.529656 6109 tablet_service.cc:1514] Processing DeleteTablet for tablet afde7296be6e4c00b71b1e30dd0c1bcd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 06:28:04 UTC) from {username='kudu', principal='kudu/127.21.12.190@KRBTEST.COM'} at 127.0.0.1:54574
I20260806 06:28:04.530030 21554 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHhLhT/build/release/bin/kudu with pid 5819
I20260806 06:28:04.538460 6468 tablet_replica.cc:330] T afde7296be6e4c00b71b1e30dd0c1bcd P 9c3bc372e32e4b9c8c4a80387367df84: stopping tablet replica
I20260806 06:28:04.538574 6468 raft_consensus.cc:2229] T afde7296be6e4c00b71b1e30dd0c1bcd P 9c3bc372e32e4b9c8c4a80387367df84 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 06:28:04.538795 6468 raft_consensus.cc:2258] T afde7296be6e4c00b71b1e30dd0c1bcd P 9c3bc372e32e4b9c8c4a80387367df84 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 06:28:04.539299 6468 ts_tablet_manager.cc:1905] T afde7296be6e4c00b71b1e30dd0c1bcd P 9c3bc372e32e4b9c8c4a80387367df84: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 06:28:04.540998 6471 tablet_replica.cc:330] T afde7296be6e4c00b71b1e30dd0c1bcd P 5bed18de14324ddebf9c7cd293d5483c: stopping tablet replica
I20260806 06:28:04.541167 6471 raft_consensus.cc:2229] T afde7296be6e4c00b71b1e30dd0c1bcd P 5bed18de14324ddebf9c7cd293d5483c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 06:28:04.541325 6471 raft_consensus.cc:2258] T afde7296be6e4c00b71b1e30dd0c1bcd P 5bed18de14324ddebf9c7cd293d5483c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 06:28:04.542394 6468 ts_tablet_manager.cc:1918] T afde7296be6e4c00b71b1e30dd0c1bcd P 9c3bc372e32e4b9c8c4a80387367df84: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 06:28:04.542479 6468 log.cc:1196] T afde7296be6e4c00b71b1e30dd0c1bcd P 9c3bc372e32e4b9c8c4a80387367df84: Deleting WAL directory at /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785997581049705-21554-0/minicluster-data/ts-2/wal/wals/afde7296be6e4c00b71b1e30dd0c1bcd
I20260806 06:28:04.543156 6468 ts_tablet_manager.cc:1939] T afde7296be6e4c00b71b1e30dd0c1bcd P 9c3bc372e32e4b9c8c4a80387367df84: Deleting consensus metadata
I20260806 06:28:04.543641 5670 catalog_manager.cc:4841] TS 9c3bc372e32e4b9c8c4a80387367df84 (127.21.12.131:37629): tablet afde7296be6e4c00b71b1e30dd0c1bcd (table test-table [id=72c88d1fe591467e94f5415a853bfd69]) successfully deleted
I20260806 06:28:04.544387 6471 ts_tablet_manager.cc:1905] T afde7296be6e4c00b71b1e30dd0c1bcd P 5bed18de14324ddebf9c7cd293d5483c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 06:28:04.546818 6471 ts_tablet_manager.cc:1918] T afde7296be6e4c00b71b1e30dd0c1bcd P 5bed18de14324ddebf9c7cd293d5483c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 06:28:04.546960 6471 log.cc:1196] T afde7296be6e4c00b71b1e30dd0c1bcd P 5bed18de14324ddebf9c7cd293d5483c: Deleting WAL directory at /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785997581049705-21554-0/minicluster-data/ts-1/wal/wals/afde7296be6e4c00b71b1e30dd0c1bcd
I20260806 06:28:04.547389 6471 ts_tablet_manager.cc:1939] T afde7296be6e4c00b71b1e30dd0c1bcd P 5bed18de14324ddebf9c7cd293d5483c: Deleting consensus metadata
I20260806 06:28:04.547879 5666 catalog_manager.cc:4841] TS 5bed18de14324ddebf9c7cd293d5483c (127.21.12.130:36787): tablet afde7296be6e4c00b71b1e30dd0c1bcd (table test-table [id=72c88d1fe591467e94f5415a853bfd69]) successfully deleted
W20260806 06:28:04.550725 5671 connection.cc:535] client connection to 127.21.12.129:43579 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260806 06:28:04.550829 5671 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260806 06:28:04.551156 21554 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHhLhT/build/release/bin/kudu with pid 6021
I20260806 06:28:04.559696 6438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:28:04.559514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.129:43579 (local address 127.0.0.1:32944)
0806 06:28:04.559570 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0806 06:28:04.559642 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.129:43579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260806 06:28:04.559847 5671 catalog_manager.cc:4584] TS be4677a62ad7407c9a534d6a8fab12b7 (127.21.12.129:43579): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet afde7296be6e4c00b71b1e30dd0c1bcd: Network error: Client connection negotiation failed: client connection to 127.21.12.129:43579: connect: Connection refused (error 111)
I20260806 06:28:04.564888 21554 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHhLhT/build/release/bin/kudu with pid 6219
I20260806 06:28:04.584187 21554 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHhLhT/build/release/bin/kudu with pid 5640
I20260806 06:28:04.593910 21554 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260806 06:28:04.937247 21554 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260806 06:28:04.937358 21554 mini_ranger.cc:66] Stopping Ranger...
I20260806 06:28:05.280920 21554 mini_ranger.cc:68] Stopped Ranger
2026-08-06 06:28:05.281 UTC [5015] LOG: received smart shutdown request
2026-08-06 06:28:05.285 UTC [5015] LOG: background worker "logical replication launcher" (PID 5030) exited with exit code 1
2026-08-06 06:28:05.285 UTC [5021] LOG: shutting down
2026-08-06 06:28:05.427 UTC [5015] LOG: database system is shut down
2026-08-06T06:28:05Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (46368 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785997581049705-21554-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 06:28:05 dist-test-slave-4tdj krb5kdc[6490](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 06:28:05 dist-test-slave-4tdj krb5kdc[6490](info): set up 2 sockets
Aug 06 06:28:05 dist-test-slave-4tdj krb5kdc[6490](info): commencing operation
krb5kdc: starting...
W20260806 06:28:06.644515 21554 mini_kdc.cc:121] Time spent starting KDC: real 1.109s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785997581049705-21554-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785997581049705-21554-0/krb5kdc/test-user.keytab.
Aug 06 06:28:06 dist-test-slave-4tdj krb5kdc[6490](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997686, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T06:28:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T06:28:06Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.21.12.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.21.12.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.21.12.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785997581049705-21554-0/krb5kdc/rangeradmin_127.21.12.148@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.21.12.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785997581049705-21554-0/krb5kdc/rangeradmin_127.21.12.148@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.21.12.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.21.12.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.21.12.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785997581049705-21554-0/krb5kdc/rangerlookup_127.21.12.148@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.21.12.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785997581049705-21554-0/krb5kdc/rangerlookup_127.21.12.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.21.12.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.21.12.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.21.12.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785997581049705-21554-0/krb5kdc/HTTP_127.21.12.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.21.12.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785997581049705-21554-0/krb5kdc/HTTP_127.21.12.148@KRBTEST.COM.keytab.
I20260806 06:28:06.797219 21554 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785997581049705-21554-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskFHhLhT/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785997581049705-21554-0/minicluster-data/postgres -l logfile start
2026-08-06 06:28:09.071 UTC [6529] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-06 06:28:09.071 UTC [6529] LOG: listening on IPv4 address "127.21.12.148", port 38817
2026-08-06 06:28:09.074 UTC [6529] LOG: listening on Unix socket "/tmp/.s.PGSQL.38817"
2026-08-06 06:28:09.093 UTC [6535] LOG: database system was shut down at 2026-08-06 06:28:07 UTC
2026-08-06 06:28:09.097 UTC [6529] LOG: database system is ready to accept connections
I20260806 06:28:10.131002 21554 mini_postgres.cc:96] Postgres bound to 38817
2026-08-06 06:28:10.137 UTC [6544] FATAL: database "slave" does not exist
127.21.12.148:38817 - accepting connections
I20260806 06:28:10.137778 21554 mini_ranger.cc:161] Starting Ranger...
I20260806 06:28:10.151548 21554 mini_ranger.cc:84] Created miniranger Postgres user
I20260806 06:28:10.552850 21554 mini_ranger.cc:87] Created ranger Postgres database
I20260806 06:28:10.552934 21554 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785997581049705-21554-0/minicluster-data/ranger-admin
2026-08-06 06:28:10,827 [I] DB FLAVOR :POSTGRES
2026-08-06 06:28:10,827 [I] --------- Verifying Ranger DB connection ---------
2026-08-06 06:28:10,828 [I] Checking connection..
2026-08-06 06:28:10,828 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFHhLhT/build/release/bin/postgresql.jar:/tmp/dist-test-taskFHhLhT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.12.148:38817/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-06 06:28:11,077 [I] Checking connection passed.
2026-08-06 06:28:11,077 [I] --------- Verifying version history table ---------
2026-08-06 06:28:11,077 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFHhLhT/build/release/bin/postgresql.jar:/tmp/dist-test-taskFHhLhT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.12.148:38817/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-06 06:28:11,335 [I] Table x_db_version_h does not exist in database ranger
2026-08-06 06:28:11,335 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFHhLhT/build/release/bin/postgresql.jar:/tmp/dist-test-taskFHhLhT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.12.148:38817/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-06 06:28:11,591 [I] Table x_db_version_h does not exist in database ranger
2026-08-06 06:28:11,591 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-06 06:28:11,591 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFHhLhT/build/release/bin/postgresql.jar:/tmp/dist-test-taskFHhLhT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.12.148:38817/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskFHhLhT/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-06 06:28:11.830 UTC [6633] WARNING: there is no transaction in progress
2026-08-06 06:28:11,837 [I] create_dbversion_catalog.sql file imported successfully
2026-08-06 06:28:11,837 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFHhLhT/build/release/bin/postgresql.jar:/tmp/dist-test-taskFHhLhT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.12.148:38817/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-06 06:28:12,099 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-06 06:28:12,099 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-06 06:28:12,100 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFHhLhT/build/release/bin/postgresql.jar:/tmp/dist-test-taskFHhLhT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.12.148:38817/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-06 06:28:12,334 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFHhLhT/build/release/bin/postgresql.jar:/tmp/dist-test-taskFHhLhT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.12.148:38817/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-06 06:28:12,564 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFHhLhT/build/release/bin/postgresql.jar:/tmp/dist-test-taskFHhLhT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.12.148:38817/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-4tdj.c.gcp-upstream.internal','N') ;"
2026-08-06 06:28:12,792 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFHhLhT/build/release/bin/postgresql.jar:/tmp/dist-test-taskFHhLhT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.12.148:38817/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-06 06:28:13,050 [I] Table x_portal_user does not exist in database ranger
2026-08-06 06:28:13,051 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFHhLhT/build/release/bin/postgresql.jar:/tmp/dist-test-taskFHhLhT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.12.148:38817/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-06 06:28:13,305 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-06 06:28:13,305 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFHhLhT/build/release/bin/postgresql.jar:/tmp/dist-test-taskFHhLhT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.12.148:38817/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-06 06:28:13,560 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFHhLhT/build/release/bin/postgresql.jar:/tmp/dist-test-taskFHhLhT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.12.148:38817/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-06 06:28:13,808 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-06 06:28:13,808 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFHhLhT/build/release/bin/postgresql.jar:/tmp/dist-test-taskFHhLhT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.12.148:38817/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskFHhLhT/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-06 06:28:14.708 UTC [6792] WARNING: there is no transaction in progress
2026-08-06 06:28:14.723 UTC [6792] WARNING: there is no transaction in progress
2026-08-06 06:28:14.739 UTC [6792] WARNING: there is no transaction in progress
2026-08-06 06:28:14.753 UTC [6792] WARNING: there is no transaction in progress
2026-08-06 06:28:14.769 UTC [6792] WARNING: there is no transaction in progress
2026-08-06 06:28:14.786 UTC [6792] WARNING: there is no transaction in progress
2026-08-06 06:28:14.844 UTC [6792] WARNING: there is no transaction in progress
2026-08-06 06:28:14.850 UTC [6792] WARNING: there is no transaction in progress
2026-08-06 06:28:14.864 UTC [6792] WARNING: there is no transaction in progress
2026-08-06 06:28:14.874 UTC [6792] WARNING: there is no transaction in progress
2026-08-06 06:28:14.884 UTC [6792] WARNING: there is no transaction in progress
2026-08-06 06:28:14.897 UTC [6792] WARNING: there is no transaction in progress
2026-08-06 06:28:14.906 UTC [6792] WARNING: there is no transaction in progress
2026-08-06 06:28:14.912 UTC [6792] WARNING: there is no transaction in progress
2026-08-06 06:28:14.918 UTC [6792] WARNING: there is no transaction in progress
2026-08-06 06:28:14.924 UTC [6792] WARNING: there is no transaction in progress
2026-08-06 06:28:15,502 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-06 06:28:15,502 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFHhLhT/build/release/bin/postgresql.jar:/tmp/dist-test-taskFHhLhT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.12.148:38817/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-06 06:28:15,734 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-06 06:28:15,734 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFHhLhT/build/release/bin/postgresql.jar:/tmp/dist-test-taskFHhLhT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.12.148:38817/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-06 06:28:15,987 [I] Table x_portal_user already exists in database 'ranger'
2026-08-06 06:28:15,987 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFHhLhT/build/release/bin/postgresql.jar:/tmp/dist-test-taskFHhLhT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.12.148:38817/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-06 06:28:16,238 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-06 06:28:16,239 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFHhLhT/build/release/bin/postgresql.jar:/tmp/dist-test-taskFHhLhT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.12.148:38817/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-06 06:28:16,493 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFHhLhT/build/release/bin/postgresql.jar:/tmp/dist-test-taskFHhLhT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.12.148:38817/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-06 06:28:16,728 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFHhLhT/build/release/bin/postgresql.jar:/tmp/dist-test-taskFHhLhT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.12.148:38817/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-4tdj.c.gcp-upstream.internal';"
2026-08-06 06:28:16,949 [I] CORE_DB_SCHEMA import status has been updated
2026-08-06 06:28:16,949 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFHhLhT/build/release/bin/postgresql.jar:/tmp/dist-test-taskFHhLhT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.12.148:38817/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-06 06:28:17,206 [I] DB_PATCHES have already been applied
I20260806 06:28:17.211476 21554 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785997581049705-21554-0/minicluster-data/ranger-admin:/tmp/dist-test-taskFHhLhT/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskFHhLhT/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskFHhLhT/build/release/bin/postgresql.jar:/tmp/dist-test-taskFHhLhT/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260806 06:28:17.211536 21554 mini_ranger.cc:193] Using host: 127.21.12.148
I20260806 06:28:17.215191 21554 mini_ranger.cc:227] Ranger admin URL: http://127.21.12.148:35997
Aug 06, 2026 6:28:17 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 06, 2026 6:28:17 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 06, 2026 6:28:17 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785997581049705-21554-0/minicluster-data/ranger-admin/ews/webapp
Aug 06, 2026 6:28:17 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785997581049705-21554-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 06, 2026 6:28:17 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785997581049705-21554-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 06, 2026 6:28:18 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785997581049705-21554-0/minicluster-data/ranger-admin/ews/webapp].
Aug 06, 2026 6:28:18 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260806 06:28:18.368606 21554 mini_ranger.cc:160] Time spent starting Ranger: real 8.231s user 0.000s sys 0.009s
Aug 06, 2026 6:28:36 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 06, 2026 6:28:37 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Aug 06, 2026 6:28:37 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 06, 2026 6:28:37 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 06, 2026 6:28:37 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 06, 2026 6:28:37 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 06, 2026 6:28:37 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 06, 2026 6:28:37 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 06, 2026 6:28:37 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 06, 2026 6:28:37 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 06, 2026 6:28:37 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 06, 2026 6:28:37 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 06, 2026 6:28:37 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 06, 2026 6:28:37 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 06, 2026 6:28:37 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 06, 2026 6:28:37 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 06, 2026 6:28:37 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 06, 2026 6:28:37 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260806 06:28:38.518884 21554 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.21.12.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.21.12.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.21.12.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785997581049705-21554-0/krb5kdc/rangerkms_127.21.12.148@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.21.12.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785997581049705-21554-0/krb5kdc/rangerkms_127.21.12.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.21.12.148@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.21.12.148@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.21.12.148@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785997581049705-21554-0/krb5kdc/HTTP_127.21.12.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.21.12.148@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785997581049705-21554-0/krb5kdc/HTTP_127.21.12.148@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 06 06:28:38 dist-test-slave-4tdj krb5kdc[6490](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997718, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260806 06:28:38.594022 21554 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260806 06:28:38.602977 21554 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260806 06:28:39.397030 21554 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260806 06:28:39.397119 21554 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785997581049705-21554-0/minicluster-data/ranger-kms
I20260806 06:28:39.397143 21554 mini_ranger_kms.cc:227] Using postgres at 127.21.12.148:38817
2026-08-06 06:28:39,609 [I] DB FLAVOR :POSTGRES
2026-08-06 06:28:39,610 [I] --------- Verifying Ranger DB connection ---------
2026-08-06 06:28:39,610 [I] Checking connection
2026-08-06 06:28:39,610 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFHhLhT/build/release/bin/postgresql.jar:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785997581049705-21554-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.12.148:38817/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-06 06:28:39,868 [I] connection success
2026-08-06 06:28:39,869 [I] --------- Verifying Ranger DB tables ---------
2026-08-06 06:28:39,869 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-06 06:28:39,869 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFHhLhT/build/release/bin/postgresql.jar:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785997581049705-21554-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.12.148:38817/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-06 06:28:40,124 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-06 06:28:40,124 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-06 06:28:40,124 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-06 06:28:40,124 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFHhLhT/build/release/bin/postgresql.jar:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785997581049705-21554-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.21.12.148:38817/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785997581049705-21554-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-06 06:28:40,391 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260806 06:28:40.834985 21554 mini_ranger_kms.cc:310] Created kms service
I20260806 06:28:40.996008 21554 mini_ranger_kms.cc:326] Created kudu user
I20260806 06:28:41.040364 21554 mini_ranger_kms.cc:343] Created rangerkms user
Aug 06 06:28:41 dist-test-slave-4tdj krb5kdc[6490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997718, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.21.12.148@KRBTEST.COM
I20260806 06:28:41.425351 21554 mini_ranger_kms.cc:384] Added ranger policy
I20260806 06:28:41.425467 21554 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785997581049705-21554-0/minicluster-data/ranger-kms:/tmp/dist-test-taskFHhLhT/build/release/bin/postgresql.jar:/tmp/dist-test-taskFHhLhT/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskFHhLhT/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260806 06:28:41.425484 21554 mini_ranger_kms.cc:240] Using host: 127.21.12.148
I20260806 06:28:41.429347 21554 mini_ranger_kms.cc:276] Ranger KMS PID: 7073
I20260806 06:28:41.429432 21554 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.21.12.148:51461
Aug 06, 2026 6:28:42 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 06, 2026 6:28:42 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 06, 2026 6:28:42 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785997581049705-21554-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 06, 2026 6:28:42 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785997581049705-21554-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 06, 2026 6:28:42 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785997581049705-21554-0/minicluster-data/ranger-kms/ews/webapp].
Aug 06, 2026 6:28:42 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Aug 06 06:28:43 dist-test-slave-4tdj krb5kdc[6490](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1785997723, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.21.12.148@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Aug 06 06:28:43 dist-test-slave-4tdj krb5kdc[6490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997723, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.21.12.148@KRBTEST.COM for HTTP/127.21.12.148@KRBTEST.COM
I20260806 06:28:43.898257 21554 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.304s user 0.002s sys 0.194s
I20260806 06:28:43.898386 21554 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260806 06:28:43.898437 21554 mini_ranger_kms.cc:401] 127.21.12.148:51461/kms/v1/keys
Aug 06 06:28:44 dist-test-slave-4tdj krb5kdc[6490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997723, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.21.12.148@KRBTEST.COM for HTTP/127.21.12.148@KRBTEST.COM
Aug 06 06:28:46 dist-test-slave-4tdj krb5kdc[6490](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997726, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.21.12.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.12.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.12.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.190.keytab.
Entry for principal kudu/127.21.12.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.190.keytab.
I20260806 06:28:46.614238 21554 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHhLhT/build/release/bin/kudu
/tmp/dist-test-taskFHhLhT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785997581049705-21554-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785997581049705-21554-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785997581049705-21554-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785997581049705-21554-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.12.190:41417
--webserver_interface=127.21.12.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.190.keytab
--principal=kudu/127.21.12.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.148:44363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.12.190:41417
--ranger_config_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785997581049705-21554-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.21.12.148:51461/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785997581049705-21554-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785997581049705-21554-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785997581049705-21554-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 06:28:46.699949 7145 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 06:28:46.700152 7145 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 06:28:46.700173 7145 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 06:28:46.701324 7145 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 06:28:46.701366 7145 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 06:28:46.701381 7145 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 06:28:46.701395 7145 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 06:28:46.701406 7145 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 06:28:46.703006 7145 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.148:44363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785997581049705-21554-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785997581049705-21554-0/minicluster-data/master-0/wal
--ranger_kms_url=127.21.12.148:51461/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.21.12.190:41417
--ranger_config_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785997581049705-21554-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.190.keytab
--principal=kudu/127.21.12.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.12.190:41417
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785997581049705-21554-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.12.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7145
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785997581049705-21554-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 05:11:02 UTC on ccycloud.quasar-hvzbbz.root.comops.site
build id 4569617
I20260806 06:28:46.703230 7145 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 06:28:46.703454 7145 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 06:28:46.708809 7153 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:28:46.708880 7145 server_base.cc:1034] running on GCE node
W20260806 06:28:46.708931 7152 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:28:46.709136 7155 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:28:46.709466 7145 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 06:28:46.709760 7145 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 06:28:46.710961 7145 hybrid_clock.cc:648] HybridClock initialized: now 1785997726710946 us; error 36 us; skew 500 ppm
Aug 06 06:28:46 dist-test-slave-4tdj krb5kdc[6490](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997726, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 06:28:46.716670 7145 init.cc:377] Logged in from keytab as kudu/127.21.12.190@KRBTEST.COM (short username kudu)
I20260806 06:28:46.717147 7145 webserver.cc:460] Webserver started at http://127.21.12.190:32995/ using document root <none> and password file <none>
I20260806 06:28:46.717347 7145 fs_manager.cc:359] Metadata directory not provided
I20260806 06:28:46.717387 7145 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 06:28:46.717511 7145 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 06 06:28:46 dist-test-slave-4tdj krb5kdc[6490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997726, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.190@KRBTEST.COM for HTTP/127.21.12.148@KRBTEST.COM
I20260806 06:28:46.764129 7145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785997581049705-21554-0/minicluster-data/master-0/data/instance:
uuid: "b92e965434714d0ba2157f809d07b69a"
format_stamp: "Formatted at 2026-08-06 06:28:46 on dist-test-slave-4tdj"
server_key: "80af98305e6a91422896484c4f00bb46"
server_key_iv: "0cdfe43166d26a230f66d35618a4f912"
server_key_version: "kuduclusterkey@0"
I20260806 06:28:46.764525 7145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785997581049705-21554-0/minicluster-data/master-0/wal/instance:
uuid: "b92e965434714d0ba2157f809d07b69a"
format_stamp: "Formatted at 2026-08-06 06:28:46 on dist-test-slave-4tdj"
server_key: "80af98305e6a91422896484c4f00bb46"
server_key_iv: "0cdfe43166d26a230f66d35618a4f912"
server_key_version: "kuduclusterkey@0"
I20260806 06:28:46.766065 7145 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260806 06:28:46.777421 7165 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 06:28:46.777732 7145 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260806 06:28:46.777858 7145 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785997581049705-21554-0/minicluster-data/master-0/data,/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785997581049705-21554-0/minicluster-data/master-0/wal
uuid: "b92e965434714d0ba2157f809d07b69a"
format_stamp: "Formatted at 2026-08-06 06:28:46 on dist-test-slave-4tdj"
server_key: "80af98305e6a91422896484c4f00bb46"
server_key_iv: "0cdfe43166d26a230f66d35618a4f912"
server_key_version: "kuduclusterkey@0"
I20260806 06:28:46.777963 7145 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785997581049705-21554-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785997581049705-21554-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785997581049705-21554-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 06:28:46.805075 7145 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 06:28:46.807040 7145 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 06:28:46.807170 7145 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 06:28:46.811908 7145 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.190:41417
I20260806 06:28:46.811973 7243 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.190:41417 every 8 connection(s)
I20260806 06:28:46.812325 7145 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785997581049705-21554-0/minicluster-data/master-0/data/info.pb
I20260806 06:28:46.813123 7244 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 06:28:46.816221 7244 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b92e965434714d0ba2157f809d07b69a: Bootstrap starting.
I20260806 06:28:46.816933 7244 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b92e965434714d0ba2157f809d07b69a: Neither blocks nor log segments found. Creating new log.
I20260806 06:28:46.817240 7244 log.cc:824] T 00000000000000000000000000000000 P b92e965434714d0ba2157f809d07b69a: Log is configured to *not* fsync() on all Append() calls
I20260806 06:28:46.818181 7244 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b92e965434714d0ba2157f809d07b69a: No bootstrap required, opened a new log
I20260806 06:28:46.820046 21554 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHhLhT/build/release/bin/kudu as pid 7145
I20260806 06:28:46.820142 21554 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785997581049705-21554-0/minicluster-data/master-0/wal/instance
I20260806 06:28:46.820319 21554 external_mini_cluster.cc:1428] Setting key aa85b21a7440bb6802bc6266652a916c
I20260806 06:28:46.820966 7244 raft_consensus.cc:348] T 00000000000000000000000000000000 P b92e965434714d0ba2157f809d07b69a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b92e965434714d0ba2157f809d07b69a" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 41417 } }
I20260806 06:28:46.821163 7244 raft_consensus.cc:374] T 00000000000000000000000000000000 P b92e965434714d0ba2157f809d07b69a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 06:28:46.821197 7244 raft_consensus.cc:729] T 00000000000000000000000000000000 P b92e965434714d0ba2157f809d07b69a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b92e965434714d0ba2157f809d07b69a, State: Initialized, Role: FOLLOWER
I20260806 06:28:46.821355 7244 consensus_queue.cc:260] T 00000000000000000000000000000000 P b92e965434714d0ba2157f809d07b69a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b92e965434714d0ba2157f809d07b69a" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 41417 } }
I20260806 06:28:46.821460 7244 raft_consensus.cc:388] T 00000000000000000000000000000000 P b92e965434714d0ba2157f809d07b69a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 06:28:46.821506 7244 raft_consensus.cc:482] T 00000000000000000000000000000000 P b92e965434714d0ba2157f809d07b69a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 06:28:46.821557 7244 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b92e965434714d0ba2157f809d07b69a [term 0 FOLLOWER]: Advancing to term 1
I20260806 06:28:46.822366 7244 raft_consensus.cc:504] T 00000000000000000000000000000000 P b92e965434714d0ba2157f809d07b69a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b92e965434714d0ba2157f809d07b69a" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 41417 } }
I20260806 06:28:46.822479 7244 leader_election.cc:302] T 00000000000000000000000000000000 P b92e965434714d0ba2157f809d07b69a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b92e965434714d0ba2157f809d07b69a; no voters:
I20260806 06:28:46.822667 7244 leader_election.cc:288] T 00000000000000000000000000000000 P b92e965434714d0ba2157f809d07b69a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 06:28:46.822717 7251 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b92e965434714d0ba2157f809d07b69a [term 1 FOLLOWER]: Leader election won for term 1
I20260806 06:28:46.822934 7251 raft_consensus.cc:686] T 00000000000000000000000000000000 P b92e965434714d0ba2157f809d07b69a [term 1 LEADER]: Becoming Leader. State: Replica: b92e965434714d0ba2157f809d07b69a, State: Running, Role: LEADER
I20260806 06:28:46.823007 7244 sys_catalog.cc:564] T 00000000000000000000000000000000 P b92e965434714d0ba2157f809d07b69a [sys.catalog]: configured and running, proceeding with master startup.
I20260806 06:28:46.823063 7251 consensus_queue.cc:237] T 00000000000000000000000000000000 P b92e965434714d0ba2157f809d07b69a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b92e965434714d0ba2157f809d07b69a" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 41417 } }
I20260806 06:28:46.824290 7251 sys_catalog.cc:455] T 00000000000000000000000000000000 P b92e965434714d0ba2157f809d07b69a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b92e965434714d0ba2157f809d07b69a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b92e965434714d0ba2157f809d07b69a" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 41417 } } }
I20260806 06:28:46.824337 7253 sys_catalog.cc:455] T 00000000000000000000000000000000 P b92e965434714d0ba2157f809d07b69a [sys.catalog]: SysCatalogTable state changed. Reason: New leader b92e965434714d0ba2157f809d07b69a. Latest consensus state: current_term: 1 leader_uuid: "b92e965434714d0ba2157f809d07b69a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b92e965434714d0ba2157f809d07b69a" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 41417 } } }
I20260806 06:28:46.824446 7253 sys_catalog.cc:458] T 00000000000000000000000000000000 P b92e965434714d0ba2157f809d07b69a [sys.catalog]: This master's current role is: LEADER
I20260806 06:28:46.824638 7251 sys_catalog.cc:458] T 00000000000000000000000000000000 P b92e965434714d0ba2157f809d07b69a [sys.catalog]: This master's current role is: LEADER
I20260806 06:28:46.824646 7244 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785997581049705-21554-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260806 06:28:46.824820 7277 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 06:28:46.825322 7277 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Aug 06 06:28:46 dist-test-slave-4tdj krb5kdc[6490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997726, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.12.190@KRBTEST.COM
I20260806 06:28:46.827564 7277 catalog_manager.cc:1347] Generated new cluster ID: 71cb50ab6ab741f4a5c8b8e27e29bf61
I20260806 06:28:46.827610 7277 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 06:28:46.832057 7249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:28:46.820824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59008 (local address 127.21.12.190:41417)
0806 06:28:46.821053 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0806 06:28:46.821062 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 06:28:46.821090 (+ 28us) server_negotiation.cc:407] Connection header received
0806 06:28:46.821194 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:28:46.821202 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:28:46.821262 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:28:46.821399 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:28:46.822109 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:28:46.823426 (+ 1317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:28:46.824373 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:28:46.824989 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:28:46.825087 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:28:46.828077 (+ 2990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:28:46.828082 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:28:46.828082 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 06:28:46.828095 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:28:46.830040 (+ 1945us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:28:46.830458 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:28:46.830462 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:28:46.830462 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:28:46.830589 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:28:46.830793 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:28:46.830794 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:28:46.830794 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:28:46.831094 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:28:46.831224 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0806 06:28:46.831731 (+ 507us) server_negotiation.cc:299] Negotiation successful
0806 06:28:46.831852 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":100,"threads_started":1}
I20260806 06:28:46.842031 7277 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 06:28:46.842902 7277 catalog_manager.cc:1504] Loading token signing keys...
I20260806 06:28:46.851764 7277 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b92e965434714d0ba2157f809d07b69a: Generated new TSK 0
I20260806 06:28:46.852165 7277 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.12.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.12.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.12.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.129.keytab.
Entry for principal kudu/127.21.12.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.129.keytab.
I20260806 06:28:47.630210 21554 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHhLhT/build/release/bin/kudu
/tmp/dist-test-taskFHhLhT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785997581049705-21554-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785997581049705-21554-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785997581049705-21554-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785997581049705-21554-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.21.12.148:51461/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.12.129:0
--local_ip_for_outbound_sockets=127.21.12.129
--webserver_interface=127.21.12.129
--webserver_port=0
--tserver_master_addrs=127.21.12.190:41417
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.129.keytab
--principal=kudu/127.21.12.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.148:44363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785997581049705-21554-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785997581049705-21554-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785997581049705-21554-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 06:28:47.749014 7320 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 06:28:47.749282 7320 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 06:28:47.749315 7320 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 06:28:47.751066 7320 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 06:28:47.751187 7320 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 06:28:47.751266 7320 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.12.129
I20260806 06:28:47.753386 7320 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.148:44363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785997581049705-21554-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785997581049705-21554-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.21.12.148:51461/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.129.keytab
--principal=kudu/127.21.12.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.12.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785997581049705-21554-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.12.129
--webserver_port=0
--tserver_master_addrs=127.21.12.190:41417
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7320
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.12.129
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785997581049705-21554-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 05:11:02 UTC on ccycloud.quasar-hvzbbz.root.comops.site
build id 4569617
I20260806 06:28:47.753769 7320 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 06:28:47.754050 7320 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 06:28:47.760308 7328 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:28:47.760421 7330 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:28:47.760669 7320 server_base.cc:1034] running on GCE node
W20260806 06:28:47.760679 7332 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:28:47.761021 7320 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 06:28:47.761278 7320 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 06:28:47.762435 7320 hybrid_clock.cc:648] HybridClock initialized: now 1785997727762412 us; error 37 us; skew 500 ppm
Aug 06 06:28:47 dist-test-slave-4tdj krb5kdc[6490](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997727, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 06:28:47.765628 7320 init.cc:377] Logged in from keytab as kudu/127.21.12.129@KRBTEST.COM (short username kudu)
I20260806 06:28:47.766114 7320 webserver.cc:460] Webserver started at http://127.21.12.129:40503/ using document root <none> and password file <none>
I20260806 06:28:47.766414 7320 fs_manager.cc:359] Metadata directory not provided
I20260806 06:28:47.766474 7320 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 06:28:47.766649 7320 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 06 06:28:47 dist-test-slave-4tdj krb5kdc[6490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997727, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.129@KRBTEST.COM for HTTP/127.21.12.148@KRBTEST.COM
I20260806 06:28:47.781221 7320 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785997581049705-21554-0/minicluster-data/ts-0/data/instance:
uuid: "8ffe730c4b924649a98da0341d0a0117"
format_stamp: "Formatted at 2026-08-06 06:28:47 on dist-test-slave-4tdj"
server_key: "d816bb72a4d351ff300a54ae3345b310"
server_key_iv: "09c6da954bfeef24e492b9701b253bc8"
server_key_version: "kuduclusterkey@0"
I20260806 06:28:47.781723 7320 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785997581049705-21554-0/minicluster-data/ts-0/wal/instance:
uuid: "8ffe730c4b924649a98da0341d0a0117"
format_stamp: "Formatted at 2026-08-06 06:28:47 on dist-test-slave-4tdj"
server_key: "d816bb72a4d351ff300a54ae3345b310"
server_key_iv: "09c6da954bfeef24e492b9701b253bc8"
server_key_version: "kuduclusterkey@0"
I20260806 06:28:47.790733 7320 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.002s sys 0.000s
I20260806 06:28:47.814399 7345 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 06:28:47.819872 7320 fs_manager.cc:725] Time spent opening block manager: real 0.014s user 0.001s sys 0.000s
I20260806 06:28:47.820003 7320 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785997581049705-21554-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785997581049705-21554-0/minicluster-data/ts-0/wal
uuid: "8ffe730c4b924649a98da0341d0a0117"
format_stamp: "Formatted at 2026-08-06 06:28:47 on dist-test-slave-4tdj"
server_key: "d816bb72a4d351ff300a54ae3345b310"
server_key_iv: "09c6da954bfeef24e492b9701b253bc8"
server_key_version: "kuduclusterkey@0"
I20260806 06:28:47.820101 7320 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785997581049705-21554-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785997581049705-21554-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785997581049705-21554-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 06:28:47.839041 7320 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 06:28:47.841517 7320 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 06:28:47.841764 7320 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 06:28:47.842016 7320 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 06:28:47.842448 7320 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 06:28:47.842502 7320 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 06:28:47.842532 7320 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 06:28:47.842585 7320 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 06:28:47.849818 7320 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.129:38729
I20260806 06:28:47.850740 7320 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785997581049705-21554-0/minicluster-data/ts-0/data/info.pb
I20260806 06:28:47.851428 21554 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHhLhT/build/release/bin/kudu as pid 7320
I20260806 06:28:47.851532 21554 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785997581049705-21554-0/minicluster-data/ts-0/wal/instance
I20260806 06:28:47.851725 21554 external_mini_cluster.cc:1428] Setting key f23c91588ef97bd51a207e84196f993a
I20260806 06:28:47.852185 7518 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.129:38729 every 8 connection(s)
Aug 06 06:28:47 dist-test-slave-4tdj krb5kdc[6490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997727, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.129@KRBTEST.COM for kudu/127.21.12.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.21.12.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.12.130@KRBTEST.COM" created.
I20260806 06:28:47.867298 7522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:28:47.851579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:41417 (local address 127.21.12.129:45587)
0806 06:28:47.852539 (+ 960us) negotiation.cc:107] Waiting for socket to connect
0806 06:28:47.852586 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0806 06:28:47.853197 (+ 611us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:28:47.853624 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:28:47.853629 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:28:47.853880 (+ 251us) client_negotiation.cc:189] Negotiated authn=SASL
0806 06:28:47.854145 (+ 265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:28:47.854151 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:28:47.855200 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:28:47.855202 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:28:47.856472 (+ 1270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:28:47.856475 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:28:47.857033 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:28:47.857035 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:28:47.857172 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:28:47.857766 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 06:28:47.857777 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0806 06:28:47.859469 (+ 1692us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 06:28:47.861517 (+ 2048us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:28:47.861524 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:28:47.861527 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:28:47.861820 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:28:47.862030 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:28:47.862032 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:28:47.862032 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:28:47.862177 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:28:47.866594 (+ 4417us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 06:28:47.866601 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 06:28:47.866775 (+ 174us) client_negotiation.cc:769] Sending connection context
0806 06:28:47.866892 (+ 117us) client_negotiation.cc:240] Negotiation successful
0806 06:28:47.867063 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":913,"thread_start_us":90,"threads_started":1}
I20260806 06:28:47.874696 7523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:28:47.851615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.129:45587 (local address 127.21.12.190:41417)
0806 06:28:47.853384 (+ 1769us) server_negotiation.cc:206] Beginning negotiation
0806 06:28:47.853386 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:28:47.853394 (+ 8us) server_negotiation.cc:407] Connection header received
0806 06:28:47.853417 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:28:47.853418 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:28:47.853431 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:28:47.853491 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:28:47.854254 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:28:47.855091 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:28:47.856567 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:28:47.856879 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:28:47.856945 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:28:47.859570 (+ 2625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:28:47.859572 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:28:47.859572 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 06:28:47.859580 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:28:47.861424 (+ 1844us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:28:47.861900 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:28:47.861902 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:28:47.861902 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:28:47.861967 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:28:47.866354 (+ 4387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:28:47.866355 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:28:47.866356 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:28:47.866516 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:28:47.866600 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0806 06:28:47.874446 (+ 7846us) server_negotiation.cc:299] Negotiation successful
0806 06:28:47.874568 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1737,"thread_start_us":75,"threads_started":1}
I20260806 06:28:47.875144 7519 heartbeater.cc:347] Connected to a master server at 127.21.12.190:41417
I20260806 06:28:47.875249 7519 heartbeater.cc:464] Registering TS with master...
I20260806 06:28:47.875476 7519 heartbeater.cc:511] Master 127.21.12.190:41417 requested a full tablet report, sending...
I20260806 06:28:47.876012 7187 ts_manager.cc:194] Registered new tserver with Master: 8ffe730c4b924649a98da0341d0a0117 (127.21.12.129:38729)
I20260806 06:28:47.876821 7187 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.12.129@KRBTEST.COM'} at 127.21.12.129:45587
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.12.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.130.keytab.
Entry for principal kudu/127.21.12.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.130.keytab.
I20260806 06:28:47.885157 21554 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHhLhT/build/release/bin/kudu
/tmp/dist-test-taskFHhLhT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785997581049705-21554-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785997581049705-21554-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785997581049705-21554-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785997581049705-21554-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.21.12.148:51461/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.12.130:0
--local_ip_for_outbound_sockets=127.21.12.130
--webserver_interface=127.21.12.130
--webserver_port=0
--tserver_master_addrs=127.21.12.190:41417
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.130.keytab
--principal=kudu/127.21.12.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.148:44363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785997581049705-21554-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785997581049705-21554-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785997581049705-21554-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 06:28:47 dist-test-slave-4tdj krb5kdc[6490](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1785997727, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260806 06:28:48.011474 7527 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 06:28:48.011818 7527 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 06:28:48.011895 7527 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 06:28:48.013626 7527 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 06:28:48.013731 7527 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 06:28:48.013818 7527 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.12.130
I20260806 06:28:48.016079 7527 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.148:44363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785997581049705-21554-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785997581049705-21554-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.21.12.148:51461/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.130.keytab
--principal=kudu/127.21.12.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.12.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785997581049705-21554-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.12.130
--webserver_port=0
--tserver_master_addrs=127.21.12.190:41417
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7527
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.12.130
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785997581049705-21554-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 05:11:02 UTC on ccycloud.quasar-hvzbbz.root.comops.site
build id 4569617
I20260806 06:28:48.016316 7527 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 06:28:48.016646 7527 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 06:28:48.022986 7540 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:28:48.023321 7537 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:28:48.023562 7527 server_base.cc:1034] running on GCE node
W20260806 06:28:48.023725 7538 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:28:48.023952 7527 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 06:28:48.024210 7527 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 06:28:48.025367 7527 hybrid_clock.cc:648] HybridClock initialized: now 1785997728025350 us; error 31 us; skew 500 ppm
Aug 06 06:28:48 dist-test-slave-4tdj krb5kdc[6490](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997728, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 06:28:48.028609 7527 init.cc:377] Logged in from keytab as kudu/127.21.12.130@KRBTEST.COM (short username kudu)
I20260806 06:28:48.029129 7527 webserver.cc:460] Webserver started at http://127.21.12.130:37713/ using document root <none> and password file <none>
I20260806 06:28:48.029398 7527 fs_manager.cc:359] Metadata directory not provided
I20260806 06:28:48.029460 7527 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 06:28:48.029713 7527 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 06 06:28:48 dist-test-slave-4tdj krb5kdc[6490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997728, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.130@KRBTEST.COM for HTTP/127.21.12.148@KRBTEST.COM
I20260806 06:28:48.052328 7527 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785997581049705-21554-0/minicluster-data/ts-1/data/instance:
uuid: "6f4e35ec5c7a4caea82350631c4e9b89"
format_stamp: "Formatted at 2026-08-06 06:28:48 on dist-test-slave-4tdj"
server_key: "f273d7b2836317573fea64b6058097d1"
server_key_iv: "143b1ec80c457c31696ce269d887cd51"
server_key_version: "kuduclusterkey@0"
I20260806 06:28:48.052845 7527 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785997581049705-21554-0/minicluster-data/ts-1/wal/instance:
uuid: "6f4e35ec5c7a4caea82350631c4e9b89"
format_stamp: "Formatted at 2026-08-06 06:28:48 on dist-test-slave-4tdj"
server_key: "f273d7b2836317573fea64b6058097d1"
server_key_iv: "143b1ec80c457c31696ce269d887cd51"
server_key_version: "kuduclusterkey@0"
I20260806 06:28:48.054807 7527 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.004s
I20260806 06:28:48.077164 7550 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 06:28:48.077399 7527 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.000s
I20260806 06:28:48.077484 7527 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785997581049705-21554-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785997581049705-21554-0/minicluster-data/ts-1/wal
uuid: "6f4e35ec5c7a4caea82350631c4e9b89"
format_stamp: "Formatted at 2026-08-06 06:28:48 on dist-test-slave-4tdj"
server_key: "f273d7b2836317573fea64b6058097d1"
server_key_iv: "143b1ec80c457c31696ce269d887cd51"
server_key_version: "kuduclusterkey@0"
I20260806 06:28:48.077577 7527 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785997581049705-21554-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785997581049705-21554-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785997581049705-21554-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 06:28:48.115156 7527 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 06:28:48.117293 7527 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 06:28:48.117449 7527 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 06:28:48.117756 7527 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 06:28:48.118144 7527 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 06:28:48.118192 7527 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 06:28:48.118222 7527 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 06:28:48.118311 7527 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 06:28:48.125784 7527 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.130:45077
I20260806 06:28:48.126315 7527 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785997581049705-21554-0/minicluster-data/ts-1/data/info.pb
I20260806 06:28:48.127579 7714 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.130:45077 every 8 connection(s)
I20260806 06:28:48.127842 21554 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHhLhT/build/release/bin/kudu as pid 7527
I20260806 06:28:48.127938 21554 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785997581049705-21554-0/minicluster-data/ts-1/wal/instance
I20260806 06:28:48.128144 21554 external_mini_cluster.cc:1428] Setting key d859fd98a9493d7d15c04e9c2faabdfb
Aug 06 06:28:48 dist-test-slave-4tdj krb5kdc[6490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997728, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.130@KRBTEST.COM for kudu/127.21.12.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.21.12.131@KRBTEST.COM; defaulting to no policy
I20260806 06:28:48.142005 7523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:28:48.129178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.130:50647 (local address 127.21.12.190:41417)
0806 06:28:48.129238 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0806 06:28:48.129240 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:28:48.131195 (+ 1955us) server_negotiation.cc:407] Connection header received
0806 06:28:48.132229 (+ 1034us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:28:48.132231 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:28:48.132247 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:28:48.132313 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:28:48.133137 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:28:48.133892 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:28:48.135416 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:28:48.135720 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:28:48.135801 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:28:48.138337 (+ 2536us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:28:48.138340 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:28:48.138340 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 06:28:48.138349 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:28:48.140382 (+ 2033us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:28:48.140894 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:28:48.140896 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:28:48.140896 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:28:48.140959 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:28:48.141235 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:28:48.141236 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:28:48.141236 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:28:48.141366 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:28:48.141411 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0806 06:28:48.141797 (+ 386us) server_negotiation.cc:299] Negotiation successful
0806 06:28:48.141891 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260806 06:28:48.142582 7718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:28:48.128909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:41417 (local address 127.21.12.130:50647)
0806 06:28:48.131044 (+ 2135us) negotiation.cc:107] Waiting for socket to connect
0806 06:28:48.131098 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0806 06:28:48.132099 (+ 1001us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:28:48.132457 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:28:48.132462 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:28:48.132731 (+ 269us) client_negotiation.cc:189] Negotiated authn=SASL
0806 06:28:48.133021 (+ 290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:28:48.133028 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:28:48.134005 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:28:48.134006 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:28:48.135303 (+ 1297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:28:48.135306 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:28:48.135833 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:28:48.135834 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:28:48.135975 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:28:48.136550 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 06:28:48.136561 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0806 06:28:48.138213 (+ 1652us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 06:28:48.140477 (+ 2264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:28:48.140484 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:28:48.140487 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:28:48.140790 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:28:48.141033 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:28:48.141033 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:28:48.141034 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:28:48.141164 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:28:48.141458 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 06:28:48.141464 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 06:28:48.141634 (+ 170us) client_negotiation.cc:769] Sending connection context
0806 06:28:48.142144 (+ 510us) client_negotiation.cc:240] Negotiation successful
0806 06:28:48.142345 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2078,"thread_start_us":82,"threads_started":1}
I20260806 06:28:48.143211 7715 heartbeater.cc:347] Connected to a master server at 127.21.12.190:41417
I20260806 06:28:48.143318 7715 heartbeater.cc:464] Registering TS with master...
I20260806 06:28:48.143532 7715 heartbeater.cc:511] Master 127.21.12.190:41417 requested a full tablet report, sending...
I20260806 06:28:48.144034 7187 ts_manager.cc:194] Registered new tserver with Master: 6f4e35ec5c7a4caea82350631c4e9b89 (127.21.12.130:45077)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.12.131@KRBTEST.COM" created.
I20260806 06:28:48.144465 7187 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.12.130@KRBTEST.COM'} at 127.21.12.130:50647
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.12.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.131.keytab.
Entry for principal kudu/127.21.12.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.131.keytab.
I20260806 06:28:48.159758 21554 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHhLhT/build/release/bin/kudu
/tmp/dist-test-taskFHhLhT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785997581049705-21554-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785997581049705-21554-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785997581049705-21554-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785997581049705-21554-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.21.12.148:51461/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.12.131:0
--local_ip_for_outbound_sockets=127.21.12.131
--webserver_interface=127.21.12.131
--webserver_port=0
--tserver_master_addrs=127.21.12.190:41417
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.131.keytab
--principal=kudu/127.21.12.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.148:44363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785997581049705-21554-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785997581049705-21554-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785997581049705-21554-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 06:28:48.254783 7723 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 06:28:48.255052 7723 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 06:28:48.255084 7723 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 06:28:48.256615 7723 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 06:28:48.256672 7723 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 06:28:48.256718 7723 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.12.131
I20260806 06:28:48.258754 7723 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.148:44363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785997581049705-21554-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785997581049705-21554-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.21.12.148:51461/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.131.keytab
--principal=kudu/127.21.12.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.12.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785997581049705-21554-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.12.131
--webserver_port=0
--tserver_master_addrs=127.21.12.190:41417
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7723
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.12.131
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785997581049705-21554-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 05:11:02 UTC on ccycloud.quasar-hvzbbz.root.comops.site
build id 4569617
I20260806 06:28:48.258985 7723 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 06:28:48.259230 7723 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 06:28:48.264400 7736 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:28:48.264387 7732 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:28:48.264585 7731 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:28:48.264747 7723 server_base.cc:1034] running on GCE node
I20260806 06:28:48.264976 7723 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 06:28:48.265277 7723 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 06:28:48.266445 7723 hybrid_clock.cc:648] HybridClock initialized: now 1785997728266421 us; error 31 us; skew 500 ppm
Aug 06 06:28:48 dist-test-slave-4tdj krb5kdc[6490](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997728, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 06:28:48.269505 7723 init.cc:377] Logged in from keytab as kudu/127.21.12.131@KRBTEST.COM (short username kudu)
I20260806 06:28:48.269896 7723 webserver.cc:460] Webserver started at http://127.21.12.131:40125/ using document root <none> and password file <none>
I20260806 06:28:48.270105 7723 fs_manager.cc:359] Metadata directory not provided
I20260806 06:28:48.270155 7723 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 06:28:48.270279 7723 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 06 06:28:48 dist-test-slave-4tdj krb5kdc[6490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997728, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.131@KRBTEST.COM for HTTP/127.21.12.148@KRBTEST.COM
I20260806 06:28:48.289290 7723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785997581049705-21554-0/minicluster-data/ts-2/data/instance:
uuid: "1dc5dd1e81394b7180fc3c4ef082769a"
format_stamp: "Formatted at 2026-08-06 06:28:48 on dist-test-slave-4tdj"
server_key: "2592164c4f4548cfb4adf847a0a7c1e2"
server_key_iv: "0829bd25a00f5832b9acfa0819e161ee"
server_key_version: "kuduclusterkey@0"
I20260806 06:28:48.289790 7723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785997581049705-21554-0/minicluster-data/ts-2/wal/instance:
uuid: "1dc5dd1e81394b7180fc3c4ef082769a"
format_stamp: "Formatted at 2026-08-06 06:28:48 on dist-test-slave-4tdj"
server_key: "2592164c4f4548cfb4adf847a0a7c1e2"
server_key_iv: "0829bd25a00f5832b9acfa0819e161ee"
server_key_version: "kuduclusterkey@0"
I20260806 06:28:48.297175 7723 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.000s sys 0.001s
I20260806 06:28:48.313746 7746 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 06:28:48.313980 7723 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260806 06:28:48.314086 7723 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785997581049705-21554-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785997581049705-21554-0/minicluster-data/ts-2/wal
uuid: "1dc5dd1e81394b7180fc3c4ef082769a"
format_stamp: "Formatted at 2026-08-06 06:28:48 on dist-test-slave-4tdj"
server_key: "2592164c4f4548cfb4adf847a0a7c1e2"
server_key_iv: "0829bd25a00f5832b9acfa0819e161ee"
server_key_version: "kuduclusterkey@0"
I20260806 06:28:48.314170 7723 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785997581049705-21554-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785997581049705-21554-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785997581049705-21554-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 06:28:48.327271 7723 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 06:28:48.338627 7723 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 06:28:48.338879 7723 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 06:28:48.339165 7723 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 06:28:48.339802 7723 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 06:28:48.339896 7723 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 06:28:48.339973 7723 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 06:28:48.340030 7723 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 06:28:48.346758 7723 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.131:45459
I20260806 06:28:48.347244 7723 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785997581049705-21554-0/minicluster-data/ts-2/data/info.pb
I20260806 06:28:48.350395 7932 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.131:45459 every 8 connection(s)
I20260806 06:28:48.356948 21554 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHhLhT/build/release/bin/kudu as pid 7723
I20260806 06:28:48.357049 21554 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785997581049705-21554-0/minicluster-data/ts-2/wal/instance
I20260806 06:28:48.357236 21554 external_mini_cluster.cc:1428] Setting key 0fb83c66656f62e59e87d26d8a8debc8
Aug 06 06:28:48 dist-test-slave-4tdj krb5kdc[6490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997728, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.131@KRBTEST.COM for kudu/127.21.12.190@KRBTEST.COM
I20260806 06:28:48.362916 7523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:28:48.351085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.131:47107 (local address 127.21.12.190:41417)
0806 06:28:48.351132 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0806 06:28:48.351134 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:28:48.351600 (+ 466us) server_negotiation.cc:407] Connection header received
0806 06:28:48.352400 (+ 800us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:28:48.352402 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:28:48.352415 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:28:48.352444 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:28:48.353287 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:28:48.354078 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:28:48.355595 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:28:48.355903 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:28:48.355950 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:28:48.359667 (+ 3717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:28:48.359669 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:28:48.359669 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 06:28:48.359677 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:28:48.361301 (+ 1624us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:28:48.361785 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:28:48.361786 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:28:48.361787 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:28:48.361847 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:28:48.362139 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:28:48.362140 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:28:48.362140 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:28:48.362271 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:28:48.362338 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0806 06:28:48.362735 (+ 397us) server_negotiation.cc:299] Negotiation successful
0806 06:28:48.362819 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260806 06:28:48.363643 7936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:28:48.351067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:41417 (local address 127.21.12.131:47107)
0806 06:28:48.351469 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0806 06:28:48.351514 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0806 06:28:48.352268 (+ 754us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:28:48.352637 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:28:48.352642 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:28:48.352899 (+ 257us) client_negotiation.cc:189] Negotiated authn=SASL
0806 06:28:48.353173 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:28:48.353179 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:28:48.354200 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:28:48.354200 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:28:48.355491 (+ 1291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:28:48.355494 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:28:48.356061 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:28:48.356062 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:28:48.356206 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:28:48.357594 (+ 1388us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 06:28:48.357604 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0806 06:28:48.359552 (+ 1948us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 06:28:48.361397 (+ 1845us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:28:48.361404 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:28:48.361406 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:28:48.361685 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:28:48.361922 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:28:48.361923 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:28:48.361923 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:28:48.362060 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:28:48.362388 (+ 328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 06:28:48.362395 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 06:28:48.362555 (+ 160us) client_negotiation.cc:769] Sending connection context
0806 06:28:48.363250 (+ 695us) client_negotiation.cc:240] Negotiation successful
0806 06:28:48.363415 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":89,"threads_started":1}
I20260806 06:28:48.364346 7933 heartbeater.cc:347] Connected to a master server at 127.21.12.190:41417
I20260806 06:28:48.364439 7933 heartbeater.cc:464] Registering TS with master...
I20260806 06:28:48.364620 7933 heartbeater.cc:511] Master 127.21.12.190:41417 requested a full tablet report, sending...
I20260806 06:28:48.365064 7187 ts_manager.cc:194] Registered new tserver with Master: 1dc5dd1e81394b7180fc3c4ef082769a (127.21.12.131:45459)
I20260806 06:28:48.365595 7187 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.12.131@KRBTEST.COM'} at 127.21.12.131:47107
I20260806 06:28:48.370030 21554 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 06:28:48.370836 21554 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHhLhT/build/release/bin/kudu with pid 7320
I20260806 06:28:48.379457 21554 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHhLhT/build/release/bin/kudu with pid 7527
I20260806 06:28:48.400142 21554 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHhLhT/build/release/bin/kudu with pid 7723
I20260806 06:28:48.410822 21554 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHhLhT/build/release/bin/kudu with pid 7145
I20260806 06:28:48.422411 21554 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHhLhT/build/release/bin/kudu
/tmp/dist-test-taskFHhLhT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785997581049705-21554-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785997581049705-21554-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785997581049705-21554-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785997581049705-21554-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.12.190:41417
--webserver_interface=127.21.12.190
--webserver_port=32995
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.190.keytab
--principal=kudu/127.21.12.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.148:44363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.12.190:41417
--ranger_config_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785997581049705-21554-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.21.12.148:51461/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.21.12.148:51461/kms with env {KRB5CCNAME=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785997581049705-21554-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785997581049705-21554-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785997581049705-21554-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 06:28:48.501988 7937 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 06:28:48.502172 7937 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 06:28:48.502202 7937 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 06:28:48.503348 7937 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 06:28:48.503391 7937 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 06:28:48.503410 7937 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 06:28:48.503433 7937 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 06:28:48.503449 7937 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 06:28:48.505134 7937 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.148:44363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785997581049705-21554-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785997581049705-21554-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.21.12.148:51461/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.21.12.190:41417
--ranger_config_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785997581049705-21554-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.190.keytab
--principal=kudu/127.21.12.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.12.190:41417
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785997581049705-21554-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.12.190
--webserver_port=32995
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7937
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785997581049705-21554-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 05:11:02 UTC on ccycloud.quasar-hvzbbz.root.comops.site
build id 4569617
I20260806 06:28:48.505373 7937 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 06:28:48.505577 7937 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260806 06:28:48.510139 7937 server_base.cc:1034] running on GCE node
W20260806 06:28:48.510092 7945 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:28:48.510072 7949 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:28:48.510068 7947 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:28:48.510563 7937 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 06:28:48.510839 7937 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 06:28:48.511967 7937 hybrid_clock.cc:648] HybridClock initialized: now 1785997728511956 us; error 28 us; skew 500 ppm
Aug 06 06:28:48 dist-test-slave-4tdj krb5kdc[6490](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997728, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 06:28:48.514182 7937 init.cc:377] Logged in from keytab as kudu/127.21.12.190@KRBTEST.COM (short username kudu)
I20260806 06:28:48.514593 7937 webserver.cc:460] Webserver started at http://127.21.12.190:32995/ using document root <none> and password file <none>
I20260806 06:28:48.514778 7937 fs_manager.cc:359] Metadata directory not provided
I20260806 06:28:48.514830 7937 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 06 06:28:48 dist-test-slave-4tdj krb5kdc[6490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997728, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.190@KRBTEST.COM for HTTP/127.21.12.148@KRBTEST.COM
I20260806 06:28:48.553368 7937 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260806 06:28:48.554071 7966 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 06:28:48.554227 7937 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260806 06:28:48.554328 7937 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785997581049705-21554-0/minicluster-data/master-0/data,/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785997581049705-21554-0/minicluster-data/master-0/wal
uuid: "b92e965434714d0ba2157f809d07b69a"
format_stamp: "Formatted at 2026-08-06 06:28:46 on dist-test-slave-4tdj"
server_key: "80af98305e6a91422896484c4f00bb46"
server_key_iv: "0cdfe43166d26a230f66d35618a4f912"
server_key_version: "kuduclusterkey@0"
I20260806 06:28:48.554680 7937 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785997581049705-21554-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785997581049705-21554-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785997581049705-21554-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 06:28:48.567242 7937 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 06:28:48.569020 7937 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 06:28:48.569130 7937 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 06:28:48.573138 7937 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.190:41417
I20260806 06:28:48.573227 8053 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.190:41417 every 8 connection(s)
I20260806 06:28:48.573559 7937 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785997581049705-21554-0/minicluster-data/master-0/data/info.pb
I20260806 06:28:48.575158 8054 sys_catalog.cc:263] Verifying existing consensus state
I20260806 06:28:48.576059 8054 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b92e965434714d0ba2157f809d07b69a: Bootstrap starting.
I20260806 06:28:48.577761 21554 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHhLhT/build/release/bin/kudu as pid 7937
I20260806 06:28:48.578008 21554 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHhLhT/build/release/bin/kudu
/tmp/dist-test-taskFHhLhT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785997581049705-21554-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785997581049705-21554-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785997581049705-21554-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785997581049705-21554-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.21.12.148:51461/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.12.129:38729
--local_ip_for_outbound_sockets=127.21.12.129
--tserver_master_addrs=127.21.12.190:41417
--webserver_port=40503
--webserver_interface=127.21.12.129
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.129.keytab
--principal=kudu/127.21.12.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.148:44363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.21.12.148:51461/kms with env {KRB5CCNAME=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785997581049705-21554-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785997581049705-21554-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785997581049705-21554-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 06:28:48.586158 8054 log.cc:824] T 00000000000000000000000000000000 P b92e965434714d0ba2157f809d07b69a: Log is configured to *not* fsync() on all Append() calls
I20260806 06:28:48.588451 8054 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b92e965434714d0ba2157f809d07b69a: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 06:28:48.588755 8054 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b92e965434714d0ba2157f809d07b69a: Bootstrap complete.
I20260806 06:28:48.592031 8054 raft_consensus.cc:348] T 00000000000000000000000000000000 P b92e965434714d0ba2157f809d07b69a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b92e965434714d0ba2157f809d07b69a" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 41417 } }
I20260806 06:28:48.592378 8054 raft_consensus.cc:729] T 00000000000000000000000000000000 P b92e965434714d0ba2157f809d07b69a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b92e965434714d0ba2157f809d07b69a, State: Initialized, Role: FOLLOWER
I20260806 06:28:48.592589 8054 consensus_queue.cc:260] T 00000000000000000000000000000000 P b92e965434714d0ba2157f809d07b69a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b92e965434714d0ba2157f809d07b69a" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 41417 } }
I20260806 06:28:48.592656 8054 raft_consensus.cc:388] T 00000000000000000000000000000000 P b92e965434714d0ba2157f809d07b69a [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 06:28:48.592697 8054 raft_consensus.cc:482] T 00000000000000000000000000000000 P b92e965434714d0ba2157f809d07b69a [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 06:28:48.592736 8054 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b92e965434714d0ba2157f809d07b69a [term 1 FOLLOWER]: Advancing to term 2
I20260806 06:28:48.593995 8054 raft_consensus.cc:504] T 00000000000000000000000000000000 P b92e965434714d0ba2157f809d07b69a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b92e965434714d0ba2157f809d07b69a" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 41417 } }
I20260806 06:28:48.594102 8054 leader_election.cc:302] T 00000000000000000000000000000000 P b92e965434714d0ba2157f809d07b69a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b92e965434714d0ba2157f809d07b69a; no voters:
I20260806 06:28:48.594257 8054 leader_election.cc:288] T 00000000000000000000000000000000 P b92e965434714d0ba2157f809d07b69a [CANDIDATE]: Term 2 election: Requested vote from peers
I20260806 06:28:48.594472 8063 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b92e965434714d0ba2157f809d07b69a [term 2 FOLLOWER]: Leader election won for term 2
I20260806 06:28:48.594518 8054 sys_catalog.cc:564] T 00000000000000000000000000000000 P b92e965434714d0ba2157f809d07b69a [sys.catalog]: configured and running, proceeding with master startup.
I20260806 06:28:48.595268 8063 raft_consensus.cc:686] T 00000000000000000000000000000000 P b92e965434714d0ba2157f809d07b69a [term 2 LEADER]: Becoming Leader. State: Replica: b92e965434714d0ba2157f809d07b69a, State: Running, Role: LEADER
I20260806 06:28:48.595422 8063 consensus_queue.cc:237] T 00000000000000000000000000000000 P b92e965434714d0ba2157f809d07b69a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b92e965434714d0ba2157f809d07b69a" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 41417 } }
I20260806 06:28:48.595953 8066 sys_catalog.cc:455] T 00000000000000000000000000000000 P b92e965434714d0ba2157f809d07b69a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "b92e965434714d0ba2157f809d07b69a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b92e965434714d0ba2157f809d07b69a" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 41417 } } }
I20260806 06:28:48.596067 8054 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785997581049705-21554-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260806 06:28:48.596088 8066 sys_catalog.cc:458] T 00000000000000000000000000000000 P b92e965434714d0ba2157f809d07b69a [sys.catalog]: This master's current role is: LEADER
I20260806 06:28:48.596321 8066 sys_catalog.cc:455] T 00000000000000000000000000000000 P b92e965434714d0ba2157f809d07b69a [sys.catalog]: SysCatalogTable state changed. Reason: New leader b92e965434714d0ba2157f809d07b69a. Latest consensus state: current_term: 2 leader_uuid: "b92e965434714d0ba2157f809d07b69a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b92e965434714d0ba2157f809d07b69a" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 41417 } } }
I20260806 06:28:48.596446 8066 sys_catalog.cc:458] T 00000000000000000000000000000000 P b92e965434714d0ba2157f809d07b69a [sys.catalog]: This master's current role is: LEADER
I20260806 06:28:48.596632 8086 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 06:28:48.599628 8086 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 06:28:48.599817 8086 catalog_manager.cc:1259] Loaded cluster ID: 71cb50ab6ab741f4a5c8b8e27e29bf61
I20260806 06:28:48.599844 8086 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 06:28:48.601667 8086 catalog_manager.cc:1504] Loading token signing keys...
I20260806 06:28:48.601900 8086 catalog_manager.cc:5879] T 00000000000000000000000000000000 P b92e965434714d0ba2157f809d07b69a: Loaded TSK: 0
I20260806 06:28:48.602150 8086 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260806 06:28:48.700248 8059 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 06:28:48.700605 8059 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 06:28:48.700698 8059 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 06:28:48.702474 8059 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 06:28:48.702682 8059 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 06:28:48.702769 8059 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.12.129
I20260806 06:28:48.704988 8059 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.148:44363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785997581049705-21554-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785997581049705-21554-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.21.12.148:51461/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.129.keytab
--principal=kudu/127.21.12.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.12.129:38729
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785997581049705-21554-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.12.129
--webserver_port=40503
--tserver_master_addrs=127.21.12.190:41417
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8059
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.12.129
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785997581049705-21554-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 05:11:02 UTC on ccycloud.quasar-hvzbbz.root.comops.site
build id 4569617
I20260806 06:28:48.705476 8059 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 06:28:48.705832 8059 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 06:28:48.712563 8115 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:28:48.712946 8113 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:28:48.713768 8117 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:28:48.713891 8059 server_base.cc:1034] running on GCE node
I20260806 06:28:48.714159 8059 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 06:28:48.714452 8059 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 06:28:48.715608 8059 hybrid_clock.cc:648] HybridClock initialized: now 1785997728715585 us; error 35 us; skew 500 ppm
Aug 06 06:28:48 dist-test-slave-4tdj krb5kdc[6490](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997728, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 06:28:48.719389 8059 init.cc:377] Logged in from keytab as kudu/127.21.12.129@KRBTEST.COM (short username kudu)
I20260806 06:28:48.719893 8059 webserver.cc:460] Webserver started at http://127.21.12.129:40503/ using document root <none> and password file <none>
I20260806 06:28:48.720172 8059 fs_manager.cc:359] Metadata directory not provided
I20260806 06:28:48.720257 8059 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 06 06:28:48 dist-test-slave-4tdj krb5kdc[6490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997728, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.129@KRBTEST.COM for HTTP/127.21.12.148@KRBTEST.COM
I20260806 06:28:48.734097 8059 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260806 06:28:48.735365 8125 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 06:28:48.735574 8059 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260806 06:28:48.735666 8059 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785997581049705-21554-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785997581049705-21554-0/minicluster-data/ts-0/wal
uuid: "8ffe730c4b924649a98da0341d0a0117"
format_stamp: "Formatted at 2026-08-06 06:28:47 on dist-test-slave-4tdj"
server_key: "d816bb72a4d351ff300a54ae3345b310"
server_key_iv: "09c6da954bfeef24e492b9701b253bc8"
server_key_version: "kuduclusterkey@0"
I20260806 06:28:48.736138 8059 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785997581049705-21554-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785997581049705-21554-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785997581049705-21554-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 06:28:48.757992 8059 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 06:28:48.760453 8059 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 06:28:48.760668 8059 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 06:28:48.760936 8059 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 06:28:48.761343 8059 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 06:28:48.761431 8059 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 06:28:48.761523 8059 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 06:28:48.761588 8059 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 06:28:48.768092 8059 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.129:38729
I20260806 06:28:48.768594 8059 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785997581049705-21554-0/minicluster-data/ts-0/data/info.pb
I20260806 06:28:48.771502 8294 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.129:38729 every 8 connection(s)
I20260806 06:28:48.772945 21554 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHhLhT/build/release/bin/kudu as pid 8059
I20260806 06:28:48.773228 21554 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHhLhT/build/release/bin/kudu
/tmp/dist-test-taskFHhLhT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785997581049705-21554-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785997581049705-21554-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785997581049705-21554-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785997581049705-21554-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.21.12.148:51461/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.12.130:45077
--local_ip_for_outbound_sockets=127.21.12.130
--tserver_master_addrs=127.21.12.190:41417
--webserver_port=37713
--webserver_interface=127.21.12.130
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.130.keytab
--principal=kudu/127.21.12.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.148:44363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.21.12.148:51461/kms with env {KRB5CCNAME=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785997581049705-21554-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785997581049705-21554-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785997581049705-21554-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 06:28:48 dist-test-slave-4tdj krb5kdc[6490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997728, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.129@KRBTEST.COM for kudu/127.21.12.190@KRBTEST.COM
I20260806 06:28:48.794896 8303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:28:48.772522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.129:60759 (local address 127.21.12.190:41417)
0806 06:28:48.779112 (+ 6590us) server_negotiation.cc:206] Beginning negotiation
0806 06:28:48.779121 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 06:28:48.779158 (+ 37us) server_negotiation.cc:407] Connection header received
0806 06:28:48.779322 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:28:48.779329 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:28:48.779387 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:28:48.779519 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:28:48.780318 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:28:48.781710 (+ 1392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:28:48.783175 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:28:48.783797 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:28:48.783901 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:28:48.790796 (+ 6895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:28:48.790802 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:28:48.790802 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 06:28:48.790815 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:28:48.792623 (+ 1808us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:28:48.793127 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:28:48.793131 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:28:48.793131 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:28:48.793257 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:28:48.793546 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:28:48.793547 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:28:48.793547 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:28:48.793854 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:28:48.793932 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0806 06:28:48.794560 (+ 628us) server_negotiation.cc:299] Negotiation successful
0806 06:28:48.794684 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6528,"thread_start_us":91,"threads_started":1}
I20260806 06:28:48.795739 8300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:28:48.772141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:41417 (local address 127.21.12.129:60759)
0806 06:28:48.778205 (+ 6064us) negotiation.cc:107] Waiting for socket to connect
0806 06:28:48.778254 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0806 06:28:48.778889 (+ 635us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:28:48.779676 (+ 787us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:28:48.779680 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:28:48.779905 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0806 06:28:48.780170 (+ 265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:28:48.780177 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:28:48.781830 (+ 1653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:28:48.781831 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:28:48.783078 (+ 1247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:28:48.783082 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:28:48.788306 (+ 5224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:28:48.788308 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:28:48.788454 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:28:48.789027 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 06:28:48.789037 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0806 06:28:48.790670 (+ 1633us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 06:28:48.792718 (+ 2048us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:28:48.792725 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:28:48.792728 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:28:48.793026 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:28:48.793339 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:28:48.793340 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:28:48.793340 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:28:48.793474 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:28:48.793963 (+ 489us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 06:28:48.793970 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 06:28:48.794118 (+ 148us) client_negotiation.cc:769] Sending connection context
0806 06:28:48.794235 (+ 117us) client_negotiation.cc:240] Negotiation successful
0806 06:28:48.794419 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6001,"thread_start_us":94,"threads_started":1}
I20260806 06:28:48.796705 8297 heartbeater.cc:347] Connected to a master server at 127.21.12.190:41417
I20260806 06:28:48.796804 8297 heartbeater.cc:464] Registering TS with master...
I20260806 06:28:48.796991 8297 heartbeater.cc:511] Master 127.21.12.190:41417 requested a full tablet report, sending...
W20260806 06:28:48.797899 8297 heartbeater.cc:644] Failed to heartbeat to 127.21.12.190:41417 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260806 06:28:48.797968 8297 heartbeater.cc:464] Registering TS with master...
I20260806 06:28:48.798027 8297 heartbeater.cc:511] Master 127.21.12.190:41417 requested a full tablet report, sending...
I20260806 06:28:48.798491 8297 heartbeater.cc:464] Registering TS with master...
I20260806 06:28:48.798563 8297 heartbeater.cc:511] Master 127.21.12.190:41417 requested a full tablet report, sending...
W20260806 06:28:48.798902 8297 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260806 06:28:48.911525 8304 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 06:28:48.911876 8304 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 06:28:48.911976 8304 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 06:28:48.913480 8304 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 06:28:48.913535 8304 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 06:28:48.913573 8304 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.12.130
I20260806 06:28:48.915098 8304 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.148:44363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785997581049705-21554-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785997581049705-21554-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.21.12.148:51461/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.130.keytab
--principal=kudu/127.21.12.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.12.130:45077
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785997581049705-21554-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.12.130
--webserver_port=37713
--tserver_master_addrs=127.21.12.190:41417
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8304
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.12.130
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785997581049705-21554-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 05:11:02 UTC on ccycloud.quasar-hvzbbz.root.comops.site
build id 4569617
I20260806 06:28:48.915331 8304 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 06:28:48.915557 8304 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260806 06:28:48.920775 8304 server_base.cc:1034] running on GCE node
W20260806 06:28:48.922483 8312 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:28:48.922479 8314 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:28:48.923110 8316 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:28:48.923398 8304 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 06:28:48.923650 8304 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 06:28:48.924816 8304 hybrid_clock.cc:648] HybridClock initialized: now 1785997728924802 us; error 29 us; skew 500 ppm
Aug 06 06:28:48 dist-test-slave-4tdj krb5kdc[6490](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997728, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 06:28:48.928095 8304 init.cc:377] Logged in from keytab as kudu/127.21.12.130@KRBTEST.COM (short username kudu)
I20260806 06:28:48.928623 8304 webserver.cc:460] Webserver started at http://127.21.12.130:37713/ using document root <none> and password file <none>
I20260806 06:28:48.928867 8304 fs_manager.cc:359] Metadata directory not provided
I20260806 06:28:48.928946 8304 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 06 06:28:48 dist-test-slave-4tdj krb5kdc[6490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997728, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.130@KRBTEST.COM for HTTP/127.21.12.148@KRBTEST.COM
I20260806 06:28:48.957312 8304 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.001s sys 0.000s
I20260806 06:28:48.958365 8325 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 06:28:48.958565 8304 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260806 06:28:48.958649 8304 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785997581049705-21554-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785997581049705-21554-0/minicluster-data/ts-1/wal
uuid: "6f4e35ec5c7a4caea82350631c4e9b89"
format_stamp: "Formatted at 2026-08-06 06:28:48 on dist-test-slave-4tdj"
server_key: "f273d7b2836317573fea64b6058097d1"
server_key_iv: "143b1ec80c457c31696ce269d887cd51"
server_key_version: "kuduclusterkey@0"
I20260806 06:28:48.959091 8304 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785997581049705-21554-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785997581049705-21554-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785997581049705-21554-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 06:28:48.975795 8304 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 06:28:48.978003 8304 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 06:28:48.978210 8304 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 06:28:48.978535 8304 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 06:28:48.978935 8304 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 06:28:48.979018 8304 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 06:28:48.979100 8304 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 06:28:48.979153 8304 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 06:28:48.985035 8304 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.130:45077
I20260806 06:28:48.985553 8304 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785997581049705-21554-0/minicluster-data/ts-1/data/info.pb
I20260806 06:28:48.990869 21554 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHhLhT/build/release/bin/kudu as pid 8304
I20260806 06:28:48.991142 21554 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHhLhT/build/release/bin/kudu
/tmp/dist-test-taskFHhLhT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785997581049705-21554-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785997581049705-21554-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785997581049705-21554-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785997581049705-21554-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.21.12.148:51461/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.12.131:45459
--local_ip_for_outbound_sockets=127.21.12.131
--tserver_master_addrs=127.21.12.190:41417
--webserver_port=40125
--webserver_interface=127.21.12.131
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.131.keytab
--principal=kudu/127.21.12.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.148:44363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.21.12.148:51461/kms with env {KRB5CCNAME=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785997581049705-21554-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785997581049705-21554-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785997581049705-21554-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 06:28:48.995956 8489 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.130:45077 every 8 connection(s)
Aug 06 06:28:49 dist-test-slave-4tdj krb5kdc[6490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997728, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.130@KRBTEST.COM for kudu/127.21.12.190@KRBTEST.COM
I20260806 06:28:49.013638 8303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:28:48.999306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.130:34421 (local address 127.21.12.190:41417)
0806 06:28:48.999355 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0806 06:28:48.999357 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:28:49.001783 (+ 2426us) server_negotiation.cc:407] Connection header received
0806 06:28:49.002587 (+ 804us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:28:49.002589 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:28:49.002608 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:28:49.002641 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:28:49.003533 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:28:49.004301 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:28:49.005842 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:28:49.006176 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:28:49.006215 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:28:49.009582 (+ 3367us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:28:49.009583 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:28:49.009584 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 06:28:49.009592 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:28:49.011235 (+ 1643us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:28:49.011701 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:28:49.011702 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:28:49.011703 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:28:49.011761 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:28:49.012446 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:28:49.012447 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:28:49.012447 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:28:49.012588 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:28:49.012630 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0806 06:28:49.013435 (+ 805us) server_negotiation.cc:299] Negotiation successful
0806 06:28:49.013535 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260806 06:28:49.015669 8495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:28:48.998991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:41417 (local address 127.21.12.130:34421)
0806 06:28:49.001619 (+ 2628us) negotiation.cc:107] Waiting for socket to connect
0806 06:28:49.001674 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0806 06:28:49.002438 (+ 764us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:28:49.002849 (+ 411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:28:49.002854 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:28:49.003110 (+ 256us) client_negotiation.cc:189] Negotiated authn=SASL
0806 06:28:49.003410 (+ 300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:28:49.003418 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:28:49.004423 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:28:49.004424 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:28:49.005742 (+ 1318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:28:49.005745 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:28:49.006340 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:28:49.006341 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:28:49.006476 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:28:49.007234 (+ 758us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 06:28:49.007244 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0806 06:28:49.009460 (+ 2216us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 06:28:49.011325 (+ 1865us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:28:49.011332 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:28:49.011335 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:28:49.011616 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:28:49.012211 (+ 595us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:28:49.012212 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:28:49.012212 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:28:49.012357 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:28:49.013097 (+ 740us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 06:28:49.013105 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 06:28:49.013273 (+ 168us) client_negotiation.cc:769] Sending connection context
0806 06:28:49.014481 (+ 1208us) client_negotiation.cc:240] Negotiation successful
0806 06:28:49.014647 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2569,"thread_start_us":93,"threads_started":1}
I20260806 06:28:49.018513 8492 heartbeater.cc:347] Connected to a master server at 127.21.12.190:41417
I20260806 06:28:49.018618 8492 heartbeater.cc:464] Registering TS with master...
I20260806 06:28:49.018821 8492 heartbeater.cc:511] Master 127.21.12.190:41417 requested a full tablet report, sending...
W20260806 06:28:49.019443 8492 heartbeater.cc:644] Failed to heartbeat to 127.21.12.190:41417 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260806 06:28:49.019528 8492 heartbeater.cc:464] Registering TS with master...
I20260806 06:28:49.019600 8492 heartbeater.cc:511] Master 127.21.12.190:41417 requested a full tablet report, sending...
I20260806 06:28:49.019881 8492 heartbeater.cc:464] Registering TS with master...
I20260806 06:28:49.019961 8492 heartbeater.cc:511] Master 127.21.12.190:41417 requested a full tablet report, sending...
W20260806 06:28:49.020251 8492 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260806 06:28:49.101557 8494 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 06:28:49.101840 8494 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 06:28:49.101883 8494 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 06:28:49.103752 8494 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 06:28:49.103813 8494 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 06:28:49.103852 8494 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.12.131
I20260806 06:28:49.106000 8494 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.148:44363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785997581049705-21554-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785997581049705-21554-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.21.12.148:51461/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.131.keytab
--principal=kudu/127.21.12.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.12.131:45459
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785997581049705-21554-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.12.131
--webserver_port=40125
--tserver_master_addrs=127.21.12.190:41417
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8494
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.12.131
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785997581049705-21554-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 05:11:02 UTC on ccycloud.quasar-hvzbbz.root.comops.site
build id 4569617
I20260806 06:28:49.106252 8494 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 06:28:49.106529 8494 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 06:28:49.113060 8505 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:28:49.113075 8504 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:28:49.114449 8508 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:28:49.117528 8494 server_base.cc:1034] running on GCE node
I20260806 06:28:49.117790 8494 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 06:28:49.118094 8494 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 06:28:49.119334 8494 hybrid_clock.cc:648] HybridClock initialized: now 1785997729119306 us; error 45 us; skew 500 ppm
Aug 06 06:28:49 dist-test-slave-4tdj krb5kdc[6490](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997729, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 06:28:49.121969 8494 init.cc:377] Logged in from keytab as kudu/127.21.12.131@KRBTEST.COM (short username kudu)
I20260806 06:28:49.122447 8494 webserver.cc:460] Webserver started at http://127.21.12.131:40125/ using document root <none> and password file <none>
I20260806 06:28:49.122681 8494 fs_manager.cc:359] Metadata directory not provided
I20260806 06:28:49.122747 8494 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 06 06:28:49 dist-test-slave-4tdj krb5kdc[6490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997729, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.131@KRBTEST.COM for HTTP/127.21.12.148@KRBTEST.COM
I20260806 06:28:49.150417 8494 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.001s
I20260806 06:28:49.151307 8521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 06:28:49.151453 8494 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 06:28:49.151573 8494 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785997581049705-21554-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785997581049705-21554-0/minicluster-data/ts-2/wal
uuid: "1dc5dd1e81394b7180fc3c4ef082769a"
format_stamp: "Formatted at 2026-08-06 06:28:48 on dist-test-slave-4tdj"
server_key: "2592164c4f4548cfb4adf847a0a7c1e2"
server_key_iv: "0829bd25a00f5832b9acfa0819e161ee"
server_key_version: "kuduclusterkey@0"
I20260806 06:28:49.151933 8494 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785997581049705-21554-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785997581049705-21554-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785997581049705-21554-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 06:28:49.171196 8494 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 06:28:49.173049 8494 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 06:28:49.173190 8494 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 06:28:49.173381 8494 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 06:28:49.173692 8494 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 06:28:49.173727 8494 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 06:28:49.173756 8494 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 06:28:49.173780 8494 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 06:28:49.179477 8494 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.131:45459
I20260806 06:28:49.179579 8676 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.131:45459 every 8 connection(s)
I20260806 06:28:49.179872 8494 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785997581049705-21554-0/minicluster-data/ts-2/data/info.pb
Aug 06 06:28:49 dist-test-slave-4tdj krb5kdc[6490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997729, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.131@KRBTEST.COM for kudu/127.21.12.190@KRBTEST.COM
I20260806 06:28:49.189018 21554 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHhLhT/build/release/bin/kudu as pid 8494
I20260806 06:28:49.190003 8303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:28:49.180667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.131:51507 (local address 127.21.12.190:41417)
0806 06:28:49.180721 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0806 06:28:49.180723 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:28:49.180886 (+ 163us) server_negotiation.cc:407] Connection header received
0806 06:28:49.181525 (+ 639us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:28:49.181526 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:28:49.181535 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:28:49.181576 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:28:49.182223 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:28:49.182842 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:28:49.183789 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:28:49.183991 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:28:49.184047 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:28:49.186520 (+ 2473us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:28:49.186522 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:28:49.186522 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 06:28:49.186531 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:28:49.188105 (+ 1574us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:28:49.188483 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:28:49.188485 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:28:49.188485 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:28:49.188561 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:28:49.188898 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:28:49.188899 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:28:49.188899 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:28:49.189040 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:28:49.189099 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0806 06:28:49.189821 (+ 722us) server_negotiation.cc:299] Negotiation successful
0806 06:28:49.189912 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260806 06:28:49.190147 8688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:28:49.180586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:41417 (local address 127.21.12.131:51507)
0806 06:28:49.180751 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0806 06:28:49.180799 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0806 06:28:49.181404 (+ 605us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:28:49.181657 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:28:49.181662 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:28:49.181873 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0806 06:28:49.182141 (+ 268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:28:49.182147 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:28:49.182925 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:28:49.182926 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:28:49.183703 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:28:49.183706 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:28:49.184098 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:28:49.184100 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:28:49.184232 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:28:49.184779 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 06:28:49.184790 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0806 06:28:49.186425 (+ 1635us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 06:28:49.188197 (+ 1772us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:28:49.188203 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:28:49.188205 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:28:49.188409 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:28:49.188651 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:28:49.188652 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:28:49.188652 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:28:49.188792 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:28:49.189126 (+ 334us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 06:28:49.189132 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 06:28:49.189656 (+ 524us) client_negotiation.cc:769] Sending connection context
0806 06:28:49.189780 (+ 124us) client_negotiation.cc:240] Negotiation successful
0806 06:28:49.189932 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":80,"threads_started":1}
I20260806 06:28:49.190781 8685 heartbeater.cc:347] Connected to a master server at 127.21.12.190:41417
I20260806 06:28:49.190871 8685 heartbeater.cc:464] Registering TS with master...
I20260806 06:28:49.191045 8685 heartbeater.cc:511] Master 127.21.12.190:41417 requested a full tablet report, sending...
W20260806 06:28:49.191668 8685 heartbeater.cc:644] Failed to heartbeat to 127.21.12.190:41417 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260806 06:28:49.191736 8685 heartbeater.cc:464] Registering TS with master...
I20260806 06:28:49.191794 8685 heartbeater.cc:511] Master 127.21.12.190:41417 requested a full tablet report, sending...
I20260806 06:28:49.192154 8685 heartbeater.cc:464] Registering TS with master...
I20260806 06:28:49.192229 8685 heartbeater.cc:511] Master 127.21.12.190:41417 requested a full tablet report, sending...
W20260806 06:28:49.192581 8685 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260806 06:28:49.199827 8689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:28:49.189392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50568 (local address 127.21.12.190:41417)
0806 06:28:49.189490 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0806 06:28:49.189492 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:28:49.192683 (+ 3191us) server_negotiation.cc:407] Connection header received
0806 06:28:49.192707 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:28:49.192708 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:28:49.192720 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:28:49.192787 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:28:49.193546 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:28:49.194364 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:28:49.195228 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:28:49.195496 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:28:49.195583 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:28:49.196791 (+ 1208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:28:49.196792 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:28:49.196792 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 06:28:49.196799 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:28:49.198472 (+ 1673us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:28:49.198931 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:28:49.198933 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:28:49.198933 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:28:49.199010 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:28:49.199245 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:28:49.199246 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:28:49.199246 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:28:49.199381 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:28:49.199567 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0806 06:28:49.199640 (+ 73us) server_negotiation.cc:299] Negotiation successful
0806 06:28:49.199725 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80,"thread_start_us":50,"threads_started":1}
Aug 06 06:28:49 dist-test-slave-4tdj krb5kdc[6490](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1785997729, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 06:28:49.799077 8297 heartbeater.cc:464] Registering TS with master...
I20260806 06:28:49.799232 8297 heartbeater.cc:511] Master 127.21.12.190:41417 requested a full tablet report, sending...
I20260806 06:28:49.799571 8001 ts_manager.cc:194] Registered new tserver with Master: 8ffe730c4b924649a98da0341d0a0117 (127.21.12.129:38729)
I20260806 06:28:49.800257 8001 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.12.129@KRBTEST.COM'} at 127.21.12.129:60759
I20260806 06:28:50.020434 8492 heartbeater.cc:464] Registering TS with master...
I20260806 06:28:50.020586 8492 heartbeater.cc:511] Master 127.21.12.190:41417 requested a full tablet report, sending...
I20260806 06:28:50.020874 8001 ts_manager.cc:194] Registered new tserver with Master: 6f4e35ec5c7a4caea82350631c4e9b89 (127.21.12.130:45077)
I20260806 06:28:50.021261 8001 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.12.130@KRBTEST.COM'} at 127.21.12.130:34421
Aug 06 06:28:50 dist-test-slave-4tdj krb5kdc[6490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997729, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.190@KRBTEST.COM for HTTP/127.21.12.148@KRBTEST.COM
I20260806 06:28:50.192754 8685 heartbeater.cc:464] Registering TS with master...
I20260806 06:28:50.192907 8685 heartbeater.cc:511] Master 127.21.12.190:41417 requested a full tablet report, sending...
I20260806 06:28:50.193250 8001 ts_manager.cc:194] Registered new tserver with Master: 1dc5dd1e81394b7180fc3c4ef082769a (127.21.12.131:45459)
I20260806 06:28:50.193768 8001 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.12.131@KRBTEST.COM'} at 127.21.12.131:51507
I20260806 06:28:50.195971 21554 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 06 06:28:50 dist-test-slave-4tdj krb5kdc[6490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997729, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.190@KRBTEST.COM for HTTP/127.21.12.148@KRBTEST.COM
W20260806 06:28:50.789554 8720 sasl_common.cc:323] Time spent creating new SASL kudu client: real 0.589s user 0.000s sys 0.000s
I20260806 06:28:50.794171 8719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:28:50.197075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50572 (local address 127.21.12.190:41417)
0806 06:28:50.197407 (+ 332us) server_negotiation.cc:206] Beginning negotiation
0806 06:28:50.197409 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:28:50.197601 (+ 192us) server_negotiation.cc:407] Connection header received
0806 06:28:50.197625 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:28:50.197625 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:28:50.197637 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:28:50.197691 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:28:50.198500 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:28:50.199268 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:28:50.200293 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:28:50.200601 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:28:50.200686 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:28:50.790711 (+590025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:28:50.790714 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:28:50.790714 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 06:28:50.790726 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:28:50.792534 (+ 1808us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:28:50.793038 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:28:50.793040 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:28:50.793041 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:28:50.793125 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:28:50.793373 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:28:50.793374 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:28:50.793374 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:28:50.793563 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:28:50.793672 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0806 06:28:50.793860 (+ 188us) server_negotiation.cc:299] Negotiation successful
0806 06:28:50.794030 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":77,"threads_started":1}
I20260806 06:28:50.796813 8001 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50572:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 06:28:50.798704 8001 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 06:28:50.801578 8297 heartbeater.cc:503] Master 127.21.12.190:41417 was elected leader, sending a full tablet report...
I20260806 06:28:50.807403 8741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:28:50.802551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.130:45077 (local address 127.0.0.1:37696)
0806 06:28:50.802796 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0806 06:28:50.803271 (+ 475us) client_negotiation.cc:174] Beginning negotiation
0806 06:28:50.803565 (+ 294us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:28:50.803835 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:28:50.803841 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:28:50.803843 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 06:28:50.803920 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:28:50.803922 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:28:50.804820 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:28:50.804821 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:28:50.805759 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:28:50.805762 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:28:50.807070 (+ 1308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:28:50.807071 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:28:50.807157 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:28:50.807172 (+ 15us) client_negotiation.cc:769] Sending connection context
0806 06:28:50.807255 (+ 83us) client_negotiation.cc:240] Negotiation successful
0806 06:28:50.807314 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":154,"threads_started":1}
I20260806 06:28:50.807438 8746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:28:50.803395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37696 (local address 127.21.12.130:45077)
0806 06:28:50.803638 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0806 06:28:50.803643 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 06:28:50.803652 (+ 9us) server_negotiation.cc:407] Connection header received
0806 06:28:50.803686 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:28:50.803688 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:28:50.803701 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:28:50.803776 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 06:28:50.803983 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:28:50.804748 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:28:50.805846 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:28:50.806705 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:28:50.806949 (+ 244us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:28:50.807037 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0806 06:28:50.807279 (+ 242us) server_negotiation.cc:299] Negotiation successful
0806 06:28:50.807325 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":211,"thread_start_us":56,"threads_started":1}
I20260806 06:28:50.808517 8395 tablet_service.cc:1467] Processing CreateTablet for tablet 550d4cdd69ce436ea038b594f6172125 (DEFAULT_TABLE table=test-table [id=93e5dd9a5727404ea5207211fdb1230a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 06:28:50.808900 8395 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 550d4cdd69ce436ea038b594f6172125. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 06:28:50.812516 8742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:28:50.802855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.129:38729 (local address 127.0.0.1:36820)
0806 06:28:50.803091 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0806 06:28:50.803109 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 06:28:50.803179 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:28:50.804016 (+ 837us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:28:50.804017 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:28:50.804018 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 06:28:50.804057 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:28:50.804058 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:28:50.805284 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:28:50.805285 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:28:50.806378 (+ 1093us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:28:50.806381 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:28:50.812263 (+ 5882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:28:50.812264 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:28:50.812330 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:28:50.812335 (+ 5us) client_negotiation.cc:769] Sending connection context
0806 06:28:50.812390 (+ 55us) client_negotiation.cc:240] Negotiation successful
0806 06:28:50.812429 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":108,"threads_started":1}
I20260806 06:28:50.813270 8741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:28:50.803083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.131:45459 (local address 127.0.0.1:38744)
0806 06:28:50.807781 (+ 4698us) negotiation.cc:107] Waiting for socket to connect
0806 06:28:50.807840 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0806 06:28:50.807888 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:28:50.808240 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:28:50.808241 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:28:50.808243 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 06:28:50.808275 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:28:50.808277 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:28:50.809981 (+ 1704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:28:50.809982 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:28:50.811967 (+ 1985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:28:50.811970 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:28:50.813040 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:28:50.813041 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:28:50.813119 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:28:50.813123 (+ 4us) client_negotiation.cc:769] Sending connection context
0806 06:28:50.813170 (+ 47us) client_negotiation.cc:240] Negotiation successful
0806 06:28:50.813208 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4687,"thread_start_us":111,"threads_started":1}
I20260806 06:28:50.813314 8745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:28:50.803172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38744 (local address 127.21.12.131:45459)
0806 06:28:50.803742 (+ 570us) server_negotiation.cc:206] Beginning negotiation
0806 06:28:50.803747 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 06:28:50.807951 (+ 4204us) server_negotiation.cc:407] Connection header received
0806 06:28:50.807997 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:28:50.807999 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:28:50.808017 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:28:50.808094 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 06:28:50.808359 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:28:50.809877 (+ 1518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:28:50.812066 (+ 2189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:28:50.812961 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:28:50.813021 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:28:50.813077 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0806 06:28:50.813190 (+ 113us) server_negotiation.cc:299] Negotiation successful
0806 06:28:50.813228 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":530,"thread_start_us":56,"threads_started":1}
I20260806 06:28:50.814241 8583 tablet_service.cc:1467] Processing CreateTablet for tablet 550d4cdd69ce436ea038b594f6172125 (DEFAULT_TABLE table=test-table [id=93e5dd9a5727404ea5207211fdb1230a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 06:28:50.814720 8583 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 550d4cdd69ce436ea038b594f6172125. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 06:28:50.814780 8743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:28:50.802947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36820 (local address 127.21.12.129:38729)
0806 06:28:50.803826 (+ 879us) server_negotiation.cc:206] Beginning negotiation
0806 06:28:50.803830 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 06:28:50.803841 (+ 11us) server_negotiation.cc:407] Connection header received
0806 06:28:50.803877 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:28:50.803879 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:28:50.803894 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:28:50.804018 (+ 124us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 06:28:50.804140 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:28:50.805167 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:28:50.811224 (+ 6057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:28:50.812189 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:28:50.812261 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:28:50.812321 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0806 06:28:50.814623 (+ 2302us) server_negotiation.cc:299] Negotiation successful
0806 06:28:50.814672 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":844,"thread_start_us":64,"threads_started":1}
I20260806 06:28:50.815575 8201 tablet_service.cc:1467] Processing CreateTablet for tablet 550d4cdd69ce436ea038b594f6172125 (DEFAULT_TABLE table=test-table [id=93e5dd9a5727404ea5207211fdb1230a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 06:28:50.815989 8201 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 550d4cdd69ce436ea038b594f6172125. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 06:28:50.818473 8747 tablet_bootstrap.cc:492] T 550d4cdd69ce436ea038b594f6172125 P 6f4e35ec5c7a4caea82350631c4e9b89: Bootstrap starting.
I20260806 06:28:50.819099 8747 tablet_bootstrap.cc:654] T 550d4cdd69ce436ea038b594f6172125 P 6f4e35ec5c7a4caea82350631c4e9b89: Neither blocks nor log segments found. Creating new log.
I20260806 06:28:50.819371 8747 log.cc:824] T 550d4cdd69ce436ea038b594f6172125 P 6f4e35ec5c7a4caea82350631c4e9b89: Log is configured to *not* fsync() on all Append() calls
I20260806 06:28:50.821498 8751 tablet_bootstrap.cc:492] T 550d4cdd69ce436ea038b594f6172125 P 8ffe730c4b924649a98da0341d0a0117: Bootstrap starting.
I20260806 06:28:50.822103 8751 tablet_bootstrap.cc:654] T 550d4cdd69ce436ea038b594f6172125 P 8ffe730c4b924649a98da0341d0a0117: Neither blocks nor log segments found. Creating new log.
I20260806 06:28:50.822434 8751 log.cc:824] T 550d4cdd69ce436ea038b594f6172125 P 8ffe730c4b924649a98da0341d0a0117: Log is configured to *not* fsync() on all Append() calls
I20260806 06:28:50.822475 8748 tablet_bootstrap.cc:492] T 550d4cdd69ce436ea038b594f6172125 P 1dc5dd1e81394b7180fc3c4ef082769a: Bootstrap starting.
I20260806 06:28:50.823146 8751 tablet_bootstrap.cc:492] T 550d4cdd69ce436ea038b594f6172125 P 8ffe730c4b924649a98da0341d0a0117: No bootstrap required, opened a new log
I20260806 06:28:50.823217 8751 ts_tablet_manager.cc:1397] T 550d4cdd69ce436ea038b594f6172125 P 8ffe730c4b924649a98da0341d0a0117: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260806 06:28:50.823688 8748 tablet_bootstrap.cc:654] T 550d4cdd69ce436ea038b594f6172125 P 1dc5dd1e81394b7180fc3c4ef082769a: Neither blocks nor log segments found. Creating new log.
I20260806 06:28:50.824760 8748 log.cc:824] T 550d4cdd69ce436ea038b594f6172125 P 1dc5dd1e81394b7180fc3c4ef082769a: Log is configured to *not* fsync() on all Append() calls
I20260806 06:28:50.825685 8751 raft_consensus.cc:348] T 550d4cdd69ce436ea038b594f6172125 P 8ffe730c4b924649a98da0341d0a0117 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f4e35ec5c7a4caea82350631c4e9b89" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 45077 } } peers { permanent_uuid: "8ffe730c4b924649a98da0341d0a0117" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 38729 } } peers { permanent_uuid: "1dc5dd1e81394b7180fc3c4ef082769a" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 45459 } }
I20260806 06:28:50.825856 8751 raft_consensus.cc:374] T 550d4cdd69ce436ea038b594f6172125 P 8ffe730c4b924649a98da0341d0a0117 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 06:28:50.825893 8751 raft_consensus.cc:729] T 550d4cdd69ce436ea038b594f6172125 P 8ffe730c4b924649a98da0341d0a0117 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ffe730c4b924649a98da0341d0a0117, State: Initialized, Role: FOLLOWER
I20260806 06:28:50.826400 8747 tablet_bootstrap.cc:492] T 550d4cdd69ce436ea038b594f6172125 P 6f4e35ec5c7a4caea82350631c4e9b89: No bootstrap required, opened a new log
I20260806 06:28:50.826479 8747 ts_tablet_manager.cc:1397] T 550d4cdd69ce436ea038b594f6172125 P 6f4e35ec5c7a4caea82350631c4e9b89: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20260806 06:28:50.829033 8747 raft_consensus.cc:348] T 550d4cdd69ce436ea038b594f6172125 P 6f4e35ec5c7a4caea82350631c4e9b89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f4e35ec5c7a4caea82350631c4e9b89" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 45077 } } peers { permanent_uuid: "8ffe730c4b924649a98da0341d0a0117" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 38729 } } peers { permanent_uuid: "1dc5dd1e81394b7180fc3c4ef082769a" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 45459 } }
I20260806 06:28:50.829192 8747 raft_consensus.cc:374] T 550d4cdd69ce436ea038b594f6172125 P 6f4e35ec5c7a4caea82350631c4e9b89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 06:28:50.829226 8747 raft_consensus.cc:729] T 550d4cdd69ce436ea038b594f6172125 P 6f4e35ec5c7a4caea82350631c4e9b89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f4e35ec5c7a4caea82350631c4e9b89, State: Initialized, Role: FOLLOWER
I20260806 06:28:50.829334 8747 consensus_queue.cc:260] T 550d4cdd69ce436ea038b594f6172125 P 6f4e35ec5c7a4caea82350631c4e9b89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f4e35ec5c7a4caea82350631c4e9b89" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 45077 } } peers { permanent_uuid: "8ffe730c4b924649a98da0341d0a0117" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 38729 } } peers { permanent_uuid: "1dc5dd1e81394b7180fc3c4ef082769a" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 45459 } }
I20260806 06:28:50.829602 8747 ts_tablet_manager.cc:1428] T 550d4cdd69ce436ea038b594f6172125 P 6f4e35ec5c7a4caea82350631c4e9b89: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260806 06:28:50.829627 8492 heartbeater.cc:503] Master 127.21.12.190:41417 was elected leader, sending a full tablet report...
I20260806 06:28:50.829727 8751 consensus_queue.cc:260] T 550d4cdd69ce436ea038b594f6172125 P 8ffe730c4b924649a98da0341d0a0117 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f4e35ec5c7a4caea82350631c4e9b89" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 45077 } } peers { permanent_uuid: "8ffe730c4b924649a98da0341d0a0117" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 38729 } } peers { permanent_uuid: "1dc5dd1e81394b7180fc3c4ef082769a" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 45459 } }
I20260806 06:28:50.830046 8751 ts_tablet_manager.cc:1428] T 550d4cdd69ce436ea038b594f6172125 P 8ffe730c4b924649a98da0341d0a0117: Time spent starting tablet: real 0.007s user 0.003s sys 0.000s
I20260806 06:28:50.839733 8748 tablet_bootstrap.cc:492] T 550d4cdd69ce436ea038b594f6172125 P 1dc5dd1e81394b7180fc3c4ef082769a: No bootstrap required, opened a new log
I20260806 06:28:50.839824 8748 ts_tablet_manager.cc:1397] T 550d4cdd69ce436ea038b594f6172125 P 1dc5dd1e81394b7180fc3c4ef082769a: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.000s
I20260806 06:28:50.843842 8748 raft_consensus.cc:348] T 550d4cdd69ce436ea038b594f6172125 P 1dc5dd1e81394b7180fc3c4ef082769a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f4e35ec5c7a4caea82350631c4e9b89" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 45077 } } peers { permanent_uuid: "8ffe730c4b924649a98da0341d0a0117" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 38729 } } peers { permanent_uuid: "1dc5dd1e81394b7180fc3c4ef082769a" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 45459 } }
I20260806 06:28:50.844054 8748 raft_consensus.cc:374] T 550d4cdd69ce436ea038b594f6172125 P 1dc5dd1e81394b7180fc3c4ef082769a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 06:28:50.844110 8748 raft_consensus.cc:729] T 550d4cdd69ce436ea038b594f6172125 P 1dc5dd1e81394b7180fc3c4ef082769a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1dc5dd1e81394b7180fc3c4ef082769a, State: Initialized, Role: FOLLOWER
I20260806 06:28:50.844246 8748 consensus_queue.cc:260] T 550d4cdd69ce436ea038b594f6172125 P 1dc5dd1e81394b7180fc3c4ef082769a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f4e35ec5c7a4caea82350631c4e9b89" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 45077 } } peers { permanent_uuid: "8ffe730c4b924649a98da0341d0a0117" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 38729 } } peers { permanent_uuid: "1dc5dd1e81394b7180fc3c4ef082769a" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 45459 } }
I20260806 06:28:50.844599 8748 ts_tablet_manager.cc:1428] T 550d4cdd69ce436ea038b594f6172125 P 1dc5dd1e81394b7180fc3c4ef082769a: Time spent starting tablet: real 0.005s user 0.000s sys 0.005s
I20260806 06:28:50.844760 8685 heartbeater.cc:503] Master 127.21.12.190:41417 was elected leader, sending a full tablet report...
W20260806 06:28:50.931239 8686 tablet.cc:2369] T 550d4cdd69ce436ea038b594f6172125 P 1dc5dd1e81394b7180fc3c4ef082769a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 06:28:50.960942 8755 raft_consensus.cc:482] T 550d4cdd69ce436ea038b594f6172125 P 6f4e35ec5c7a4caea82350631c4e9b89 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 06:28:50.961087 8755 raft_consensus.cc:504] T 550d4cdd69ce436ea038b594f6172125 P 6f4e35ec5c7a4caea82350631c4e9b89 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f4e35ec5c7a4caea82350631c4e9b89" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 45077 } } peers { permanent_uuid: "8ffe730c4b924649a98da0341d0a0117" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 38729 } } peers { permanent_uuid: "1dc5dd1e81394b7180fc3c4ef082769a" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 45459 } }
I20260806 06:28:50.961549 8755 leader_election.cc:288] T 550d4cdd69ce436ea038b594f6172125 P 6f4e35ec5c7a4caea82350631c4e9b89 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8ffe730c4b924649a98da0341d0a0117 (127.21.12.129:38729), 1dc5dd1e81394b7180fc3c4ef082769a (127.21.12.131:45459)
I20260806 06:28:50.965826 8743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:28:50.961910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.130:54865 (local address 127.21.12.129:38729)
0806 06:28:50.961975 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0806 06:28:50.961977 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:28:50.962094 (+ 117us) server_negotiation.cc:407] Connection header received
0806 06:28:50.962218 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:28:50.962219 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:28:50.962230 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:28:50.962262 (+ 32us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 06:28:50.962520 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:28:50.963159 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:28:50.964842 (+ 1683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:28:50.965409 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:28:50.965477 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:28:50.965485 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0806 06:28:50.965710 (+ 225us) server_negotiation.cc:299] Negotiation successful
0806 06:28:50.965761 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260806 06:28:50.965842 8745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:28:50.962152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.130:39677 (local address 127.21.12.131:45459)
0806 06:28:50.962199 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0806 06:28:50.962202 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 06:28:50.962394 (+ 192us) server_negotiation.cc:407] Connection header received
0806 06:28:50.962472 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:28:50.962474 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:28:50.962485 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:28:50.962540 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 06:28:50.962737 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:28:50.963472 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:28:50.964916 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:28:50.965358 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:28:50.965425 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:28:50.965434 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0806 06:28:50.965743 (+ 309us) server_negotiation.cc:299] Negotiation successful
0806 06:28:50.965785 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260806 06:28:50.965831 8759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:28:50.961752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.129:38729 (local address 127.21.12.130:54865)
0806 06:28:50.962022 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0806 06:28:50.962034 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 06:28:50.962127 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:28:50.962370 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:28:50.962371 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:28:50.962373 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 06:28:50.962454 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:28:50.962457 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:28:50.963236 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:28:50.963238 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:28:50.964751 (+ 1513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:28:50.964754 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:28:50.965542 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:28:50.965544 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:28:50.965627 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:28:50.965633 (+ 6us) client_negotiation.cc:769] Sending connection context
0806 06:28:50.965688 (+ 55us) client_negotiation.cc:240] Negotiation successful
0806 06:28:50.965728 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":187,"threads_started":1}
I20260806 06:28:50.966212 8760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:28:50.962091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.131:45459 (local address 127.21.12.130:39677)
0806 06:28:50.962337 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0806 06:28:50.962347 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0806 06:28:50.962396 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:28:50.962540 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:28:50.962541 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:28:50.962542 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 06:28:50.962667 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:28:50.962669 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:28:50.963570 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:28:50.963572 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:28:50.964811 (+ 1239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:28:50.964813 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:28:50.965555 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:28:50.965557 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:28:50.965650 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:28:50.965656 (+ 6us) client_negotiation.cc:769] Sending connection context
0806 06:28:50.965945 (+ 289us) client_negotiation.cc:240] Negotiation successful
0806 06:28:50.965987 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":118,"threads_started":1}
I20260806 06:28:50.966425 8227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "550d4cdd69ce436ea038b594f6172125" candidate_uuid: "6f4e35ec5c7a4caea82350631c4e9b89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ffe730c4b924649a98da0341d0a0117" is_pre_election: true
I20260806 06:28:50.966593 8227 raft_consensus.cc:2454] T 550d4cdd69ce436ea038b594f6172125 P 8ffe730c4b924649a98da0341d0a0117 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6f4e35ec5c7a4caea82350631c4e9b89 in term 0.
I20260806 06:28:50.966732 8616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "550d4cdd69ce436ea038b594f6172125" candidate_uuid: "6f4e35ec5c7a4caea82350631c4e9b89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dc5dd1e81394b7180fc3c4ef082769a" is_pre_election: true
I20260806 06:28:50.966864 8329 leader_election.cc:302] T 550d4cdd69ce436ea038b594f6172125 P 6f4e35ec5c7a4caea82350631c4e9b89 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6f4e35ec5c7a4caea82350631c4e9b89, 8ffe730c4b924649a98da0341d0a0117; no voters:
I20260806 06:28:50.966969 8616 raft_consensus.cc:2454] T 550d4cdd69ce436ea038b594f6172125 P 1dc5dd1e81394b7180fc3c4ef082769a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6f4e35ec5c7a4caea82350631c4e9b89 in term 0.
I20260806 06:28:50.966970 8755 raft_consensus.cc:2781] T 550d4cdd69ce436ea038b594f6172125 P 6f4e35ec5c7a4caea82350631c4e9b89 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 06:28:50.967032 8755 raft_consensus.cc:482] T 550d4cdd69ce436ea038b594f6172125 P 6f4e35ec5c7a4caea82350631c4e9b89 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 06:28:50.967065 8755 raft_consensus.cc:3037] T 550d4cdd69ce436ea038b594f6172125 P 6f4e35ec5c7a4caea82350631c4e9b89 [term 0 FOLLOWER]: Advancing to term 1
I20260806 06:28:50.968108 8755 raft_consensus.cc:504] T 550d4cdd69ce436ea038b594f6172125 P 6f4e35ec5c7a4caea82350631c4e9b89 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f4e35ec5c7a4caea82350631c4e9b89" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 45077 } } peers { permanent_uuid: "8ffe730c4b924649a98da0341d0a0117" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 38729 } } peers { permanent_uuid: "1dc5dd1e81394b7180fc3c4ef082769a" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 45459 } }
I20260806 06:28:50.968214 8755 leader_election.cc:288] T 550d4cdd69ce436ea038b594f6172125 P 6f4e35ec5c7a4caea82350631c4e9b89 [CANDIDATE]: Term 1 election: Requested vote from peers 8ffe730c4b924649a98da0341d0a0117 (127.21.12.129:38729), 1dc5dd1e81394b7180fc3c4ef082769a (127.21.12.131:45459)
I20260806 06:28:50.968412 8227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "550d4cdd69ce436ea038b594f6172125" candidate_uuid: "6f4e35ec5c7a4caea82350631c4e9b89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ffe730c4b924649a98da0341d0a0117"
I20260806 06:28:50.968508 8227 raft_consensus.cc:3037] T 550d4cdd69ce436ea038b594f6172125 P 8ffe730c4b924649a98da0341d0a0117 [term 0 FOLLOWER]: Advancing to term 1
I20260806 06:28:50.968559 8616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "550d4cdd69ce436ea038b594f6172125" candidate_uuid: "6f4e35ec5c7a4caea82350631c4e9b89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dc5dd1e81394b7180fc3c4ef082769a"
I20260806 06:28:50.968642 8616 raft_consensus.cc:3037] T 550d4cdd69ce436ea038b594f6172125 P 1dc5dd1e81394b7180fc3c4ef082769a [term 0 FOLLOWER]: Advancing to term 1
I20260806 06:28:50.969440 8227 raft_consensus.cc:2454] T 550d4cdd69ce436ea038b594f6172125 P 8ffe730c4b924649a98da0341d0a0117 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6f4e35ec5c7a4caea82350631c4e9b89 in term 1.
I20260806 06:28:50.969632 8329 leader_election.cc:302] T 550d4cdd69ce436ea038b594f6172125 P 6f4e35ec5c7a4caea82350631c4e9b89 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6f4e35ec5c7a4caea82350631c4e9b89, 8ffe730c4b924649a98da0341d0a0117; no voters:
I20260806 06:28:50.969645 8616 raft_consensus.cc:2454] T 550d4cdd69ce436ea038b594f6172125 P 1dc5dd1e81394b7180fc3c4ef082769a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6f4e35ec5c7a4caea82350631c4e9b89 in term 1.
I20260806 06:28:50.969746 8755 raft_consensus.cc:2781] T 550d4cdd69ce436ea038b594f6172125 P 6f4e35ec5c7a4caea82350631c4e9b89 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 06:28:50.969902 8755 raft_consensus.cc:686] T 550d4cdd69ce436ea038b594f6172125 P 6f4e35ec5c7a4caea82350631c4e9b89 [term 1 LEADER]: Becoming Leader. State: Replica: 6f4e35ec5c7a4caea82350631c4e9b89, State: Running, Role: LEADER
I20260806 06:28:50.970000 8755 consensus_queue.cc:237] T 550d4cdd69ce436ea038b594f6172125 P 6f4e35ec5c7a4caea82350631c4e9b89 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f4e35ec5c7a4caea82350631c4e9b89" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 45077 } } peers { permanent_uuid: "8ffe730c4b924649a98da0341d0a0117" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 38729 } } peers { permanent_uuid: "1dc5dd1e81394b7180fc3c4ef082769a" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 45459 } }
I20260806 06:28:50.970893 7996 catalog_manager.cc:5495] T 550d4cdd69ce436ea038b594f6172125 P 6f4e35ec5c7a4caea82350631c4e9b89 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6f4e35ec5c7a4caea82350631c4e9b89 (127.21.12.130). New cstate: current_term: 1 leader_uuid: "6f4e35ec5c7a4caea82350631c4e9b89" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f4e35ec5c7a4caea82350631c4e9b89" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 45077 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8ffe730c4b924649a98da0341d0a0117" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 38729 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1dc5dd1e81394b7180fc3c4ef082769a" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 45459 } health_report { overall_health: UNKNOWN } } }
W20260806 06:28:50.999895 8493 tablet.cc:2369] T 550d4cdd69ce436ea038b594f6172125 P 6f4e35ec5c7a4caea82350631c4e9b89: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 06:28:51.007201 8746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:28:51.005088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37712 (local address 127.21.12.130:45077)
0806 06:28:51.005143 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0806 06:28:51.005144 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0806 06:28:51.005151 (+ 7us) server_negotiation.cc:407] Connection header received
0806 06:28:51.005180 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:28:51.005181 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:28:51.005196 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:28:51.005264 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 06:28:51.005368 (+ 104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:28:51.005821 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:28:51.006601 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:28:51.006777 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:28:51.006818 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:28:51.006957 (+ 139us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 06:28:51.007046 (+ 89us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 06:28:51.007111 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0806 06:28:51.007122 (+ 11us) server_negotiation.cc:299] Negotiation successful
0806 06:28:51.007138 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260806 06:28:51.009133 8616 raft_consensus.cc:1261] T 550d4cdd69ce436ea038b594f6172125 P 1dc5dd1e81394b7180fc3c4ef082769a [term 1 FOLLOWER]: Refusing update from remote peer 6f4e35ec5c7a4caea82350631c4e9b89: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 06:28:51.009142 8227 raft_consensus.cc:1261] T 550d4cdd69ce436ea038b594f6172125 P 8ffe730c4b924649a98da0341d0a0117 [term 1 FOLLOWER]: Refusing update from remote peer 6f4e35ec5c7a4caea82350631c4e9b89: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 06:28:51.009466 8762 consensus_queue.cc:1035] T 550d4cdd69ce436ea038b594f6172125 P 6f4e35ec5c7a4caea82350631c4e9b89 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ffe730c4b924649a98da0341d0a0117" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 38729 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 06:28:51.009591 8755 consensus_queue.cc:1035] T 550d4cdd69ce436ea038b594f6172125 P 6f4e35ec5c7a4caea82350631c4e9b89 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dc5dd1e81394b7180fc3c4ef082769a" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 45459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 06:28:51.012282 8766 mvcc.cc:205] Tried to move back new op lower bound from 7315446706210152448 to 7315446706053672960. Current Snapshot: MvccSnapshot[applied={T|T < 7315446706210152448}]
I20260806 06:28:51.012815 8767 mvcc.cc:205] Tried to move back new op lower bound from 7315446706210152448 to 7315446706053672960. Current Snapshot: MvccSnapshot[applied={T|T < 7315446706210152448}]
I20260806 06:28:51.014362 7996 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50572:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 06:28:51.014446 7996 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50572:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 06:28:51.015151 7996 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b92e965434714d0ba2157f809d07b69a: Sending DeleteTablet for 3 replicas of tablet 550d4cdd69ce436ea038b594f6172125
I20260806 06:28:51.015535 8395 tablet_service.cc:1514] Processing DeleteTablet for tablet 550d4cdd69ce436ea038b594f6172125 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 06:28:51 UTC) from {username='kudu', principal='kudu/127.21.12.190@KRBTEST.COM'} at 127.0.0.1:37696
I20260806 06:28:51.015658 8583 tablet_service.cc:1514] Processing DeleteTablet for tablet 550d4cdd69ce436ea038b594f6172125 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 06:28:51 UTC) from {username='kudu', principal='kudu/127.21.12.190@KRBTEST.COM'} at 127.0.0.1:38744
I20260806 06:28:51.015758 8201 tablet_service.cc:1514] Processing DeleteTablet for tablet 550d4cdd69ce436ea038b594f6172125 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 06:28:51 UTC) from {username='kudu', principal='kudu/127.21.12.190@KRBTEST.COM'} at 127.0.0.1:36820
I20260806 06:28:51.015961 8781 tablet_replica.cc:330] T 550d4cdd69ce436ea038b594f6172125 P 8ffe730c4b924649a98da0341d0a0117: stopping tablet replica
I20260806 06:28:51.016060 8781 raft_consensus.cc:2229] T 550d4cdd69ce436ea038b594f6172125 P 8ffe730c4b924649a98da0341d0a0117 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 06:28:51.016170 8781 raft_consensus.cc:2258] T 550d4cdd69ce436ea038b594f6172125 P 8ffe730c4b924649a98da0341d0a0117 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 06:28:51.016163 8779 tablet_replica.cc:330] T 550d4cdd69ce436ea038b594f6172125 P 1dc5dd1e81394b7180fc3c4ef082769a: stopping tablet replica
I20260806 06:28:51.016222 21554 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHhLhT/build/release/bin/kudu with pid 8059
I20260806 06:28:51.016305 8779 raft_consensus.cc:2229] T 550d4cdd69ce436ea038b594f6172125 P 1dc5dd1e81394b7180fc3c4ef082769a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 06:28:51.016386 8779 raft_consensus.cc:2258] T 550d4cdd69ce436ea038b594f6172125 P 1dc5dd1e81394b7180fc3c4ef082769a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 06:28:51.023294 8779 ts_tablet_manager.cc:1905] T 550d4cdd69ce436ea038b594f6172125 P 1dc5dd1e81394b7180fc3c4ef082769a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 06:28:51.025439 8779 ts_tablet_manager.cc:1918] T 550d4cdd69ce436ea038b594f6172125 P 1dc5dd1e81394b7180fc3c4ef082769a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 06:28:51.025518 8779 log.cc:1196] T 550d4cdd69ce436ea038b594f6172125 P 1dc5dd1e81394b7180fc3c4ef082769a: Deleting WAL directory at /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785997581049705-21554-0/minicluster-data/ts-2/wal/wals/550d4cdd69ce436ea038b594f6172125
I20260806 06:28:51.025794 8779 ts_tablet_manager.cc:1939] T 550d4cdd69ce436ea038b594f6172125 P 1dc5dd1e81394b7180fc3c4ef082769a: Deleting consensus metadata
I20260806 06:28:51.026227 7970 catalog_manager.cc:4841] TS 1dc5dd1e81394b7180fc3c4ef082769a (127.21.12.131:45459): tablet 550d4cdd69ce436ea038b594f6172125 (table test-table [id=93e5dd9a5727404ea5207211fdb1230a]) successfully deleted
W20260806 06:28:51.027144 7970 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.12.129:38729 (error 108)
I20260806 06:28:51.027427 21554 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHhLhT/build/release/bin/kudu with pid 8304
I20260806 06:28:51.028400 8741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:28:51.027487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.129:38729 (local address 127.0.0.1:36834)
0806 06:28:51.028300 (+ 813us) negotiation.cc:107] Waiting for socket to connect
0806 06:28:51.028363 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.129:38729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":786}
W20260806 06:28:51.029402 7970 catalog_manager.cc:4584] TS 8ffe730c4b924649a98da0341d0a0117 (127.21.12.129:38729): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 550d4cdd69ce436ea038b594f6172125: Network error: Client connection negotiation failed: client connection to 127.21.12.129:38729: connect: Connection refused (error 111)
I20260806 06:28:51.049450 8741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:28:51.049322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.130:45077 (local address 127.0.0.1:37724)
0806 06:28:51.049367 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0806 06:28:51.049400 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.130:45077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
I20260806 06:28:51.049665 21554 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHhLhT/build/release/bin/kudu with pid 8494
I20260806 06:28:51.060137 21554 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHhLhT/build/release/bin/kudu with pid 7937
I20260806 06:28:51.071789 21554 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260806 06:28:51.417572 21554 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260806 06:28:51.417671 21554 mini_ranger.cc:66] Stopping Ranger...
I20260806 06:28:51.761475 21554 mini_ranger.cc:68] Stopped Ranger
2026-08-06 06:28:51.761 UTC [6529] LOG: received smart shutdown request
2026-08-06 06:28:51.766 UTC [6529] LOG: background worker "logical replication launcher" (PID 6542) exited with exit code 1
2026-08-06 06:28:51.766 UTC [6536] LOG: shutting down
2026-08-06 06:28:51.903 UTC [6529] LOG: database system is shut down
2026-08-06T06:28:51Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (46482 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785997581049705-21554-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 06:28:52 dist-test-slave-4tdj krb5kdc[8794](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 06:28:52 dist-test-slave-4tdj krb5kdc[8794](info): set up 2 sockets
Aug 06 06:28:52 dist-test-slave-4tdj krb5kdc[8794](info): commencing operation
krb5kdc: starting...
W20260806 06:28:53.123411 21554 mini_kdc.cc:121] Time spent starting KDC: real 1.106s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785997581049705-21554-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785997581049705-21554-0/krb5kdc/test-user.keytab.
Aug 06 06:28:53 dist-test-slave-4tdj krb5kdc[8794](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997733, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T06:28:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T06:28:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.12.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.12.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.12.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.190.keytab.
Entry for principal kudu/127.21.12.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.190.keytab.
I20260806 06:28:53.234858 21554 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHhLhT/build/release/bin/kudu
/tmp/dist-test-taskFHhLhT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785997581049705-21554-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785997581049705-21554-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785997581049705-21554-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785997581049705-21554-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.12.190:40733
--webserver_interface=127.21.12.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.190.keytab
--principal=kudu/127.21.12.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.148:44185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.12.190:40733,127.21.12.189:46377,127.21.12.188:39245
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785997581049705-21554-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785997581049705-21554-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785997581049705-21554-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 06:28:53.313956 8816 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 06:28:53.314170 8816 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 06:28:53.314191 8816 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 06:28:53.315363 8816 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 06:28:53.315407 8816 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 06:28:53.315418 8816 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 06:28:53.315431 8816 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 06:28:53.315454 8816 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 06:28:53.316998 8816 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.148:44185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785997581049705-21554-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785997581049705-21554-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.12.190:40733,127.21.12.189:46377,127.21.12.188:39245
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.190.keytab
--principal=kudu/127.21.12.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.12.190:40733
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785997581049705-21554-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.12.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8816
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785997581049705-21554-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 05:11:02 UTC on ccycloud.quasar-hvzbbz.root.comops.site
build id 4569617
I20260806 06:28:53.317242 8816 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 06:28:53.317448 8816 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 06:28:53.322052 8829 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:28:53.322201 8816 server_base.cc:1034] running on GCE node
W20260806 06:28:53.322141 8825 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:28:53.322078 8826 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:28:53.322587 8816 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 06:28:53.322847 8816 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 06:28:53.323998 8816 hybrid_clock.cc:648] HybridClock initialized: now 1785997733323976 us; error 36 us; skew 500 ppm
Aug 06 06:28:53 dist-test-slave-4tdj krb5kdc[8794](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997733, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 06:28:53.326457 8816 init.cc:377] Logged in from keytab as kudu/127.21.12.190@KRBTEST.COM (short username kudu)
I20260806 06:28:53.326848 8816 webserver.cc:460] Webserver started at http://127.21.12.190:38881/ using document root <none> and password file <none>
I20260806 06:28:53.327062 8816 fs_manager.cc:359] Metadata directory not provided
I20260806 06:28:53.327108 8816 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 06:28:53.327239 8816 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 06:28:53.328229 8816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785997581049705-21554-0/minicluster-data/master-0/data/instance:
uuid: "81239de6d9b14d648ebebd041840a733"
format_stamp: "Formatted at 2026-08-06 06:28:53 on dist-test-slave-4tdj"
server_key: "c88e530810ec7defa834daa6ff75dafb"
server_key_iv: "2c845db3fe02a76b05cfb0504089a7aa"
server_key_version: "encryptionkey@0"
I20260806 06:28:53.328565 8816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785997581049705-21554-0/minicluster-data/master-0/wal/instance:
uuid: "81239de6d9b14d648ebebd041840a733"
format_stamp: "Formatted at 2026-08-06 06:28:53 on dist-test-slave-4tdj"
server_key: "c88e530810ec7defa834daa6ff75dafb"
server_key_iv: "2c845db3fe02a76b05cfb0504089a7aa"
server_key_version: "encryptionkey@0"
I20260806 06:28:53.329804 8816 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260806 06:28:53.330538 8838 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 06:28:53.330682 8816 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260806 06:28:53.330767 8816 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785997581049705-21554-0/minicluster-data/master-0/data,/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785997581049705-21554-0/minicluster-data/master-0/wal
uuid: "81239de6d9b14d648ebebd041840a733"
format_stamp: "Formatted at 2026-08-06 06:28:53 on dist-test-slave-4tdj"
server_key: "c88e530810ec7defa834daa6ff75dafb"
server_key_iv: "2c845db3fe02a76b05cfb0504089a7aa"
server_key_version: "encryptionkey@0"
I20260806 06:28:53.330827 8816 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785997581049705-21554-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785997581049705-21554-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785997581049705-21554-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 06:28:53.339761 8816 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 06:28:53.341465 8816 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 06:28:53.341591 8816 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 06:28:53.345225 8816 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.190:40733
I20260806 06:28:53.345275 8912 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.190:40733 every 8 connection(s)
I20260806 06:28:53.345568 8816 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785997581049705-21554-0/minicluster-data/master-0/data/info.pb
I20260806 06:28:53.346452 8914 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 06:28:53.348045 8914 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.12.190" port: 40733 } has no permanent_uuid. Determining permanent_uuid...
I20260806 06:28:53.348914 21554 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHhLhT/build/release/bin/kudu as pid 8816
I20260806 06:28:53.349002 21554 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785997581049705-21554-0/minicluster-data/master-0/wal/instance
I20260806 06:28:53.349164 21554 external_mini_cluster.cc:1428] Setting key e2a479223ac657c5821ef08cd55ff0d1
Aug 06 06:28:53 dist-test-slave-4tdj krb5kdc[8794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997733, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.190@KRBTEST.COM for kudu/127.21.12.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.21.12.189@KRBTEST.COM; defaulting to no policy
I20260806 06:28:53.357259 8916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:28:53.348657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:40733 (local address 127.0.0.1:54310)
0806 06:28:53.348820 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0806 06:28:53.348860 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0806 06:28:53.349387 (+ 527us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:28:53.349664 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:28:53.349667 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:28:53.349936 (+ 269us) client_negotiation.cc:189] Negotiated authn=SASL
0806 06:28:53.350093 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:28:53.350097 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:28:53.351158 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:28:53.351159 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:28:53.351830 (+ 671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:28:53.351833 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:28:53.352323 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:28:53.352324 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:28:53.352395 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:28:53.352880 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 06:28:53.352891 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0806 06:28:53.354426 (+ 1535us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 06:28:53.355977 (+ 1551us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:28:53.355983 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:28:53.355985 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:28:53.356157 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:28:53.356355 (+ 198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:28:53.356356 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:28:53.356356 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:28:53.356435 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:28:53.356740 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 06:28:53.356745 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 06:28:53.356802 (+ 57us) client_negotiation.cc:769] Sending connection context
0806 06:28:53.356880 (+ 78us) client_negotiation.cc:240] Negotiation successful
0806 06:28:53.357081 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"mutex_wait_us":75,"thread_start_us":66,"threads_started":1}
I20260806 06:28:53.357264 8917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:28:53.348689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54310 (local address 127.21.12.190:40733)
0806 06:28:53.348860 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0806 06:28:53.348869 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 06:28:53.348957 (+ 88us) server_negotiation.cc:407] Connection header received
0806 06:28:53.349548 (+ 591us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:28:53.349550 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:28:53.349571 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:28:53.349652 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:28:53.350187 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:28:53.351063 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:28:53.351914 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:28:53.352243 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:28:53.352354 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:28:53.354511 (+ 2157us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:28:53.354517 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:28:53.354517 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 06:28:53.354523 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:28:53.355901 (+ 1378us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:28:53.356226 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:28:53.356230 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:28:53.356230 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:28:53.356300 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:28:53.356486 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:28:53.356487 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:28:53.356488 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:28:53.356679 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:28:53.356748 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0806 06:28:53.356910 (+ 162us) server_negotiation.cc:299] Negotiation successful
0806 06:28:53.357010 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":78,"threads_started":1}
I20260806 06:28:53.357975 8914 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.12.189" port: 46377 } has no permanent_uuid. Determining permanent_uuid...
I20260806 06:28:53.358325 8916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:28:53.358183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.189:46377 (local address 127.0.0.1:36770)
0806 06:28:53.358229 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0806 06:28:53.358278 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.189:46377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260806 06:28:53.358455 8841 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.12.189:46377: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.12.189@KRBTEST.COM" created.
I20260806 06:28:53.358867 8916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:28:53.358785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.189:46377 (local address 127.0.0.1:36776)
0806 06:28:53.358818 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0806 06:28:53.358845 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.189:46377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
W20260806 06:28:53.358965 8914 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.12.189:46377: Network error: Client connection negotiation failed: client connection to 127.21.12.189:46377: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.12.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.189.keytab.
Entry for principal kudu/127.21.12.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.189.keytab.
I20260806 06:28:53.371279 21554 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHhLhT/build/release/bin/kudu
/tmp/dist-test-taskFHhLhT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785997581049705-21554-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785997581049705-21554-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785997581049705-21554-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785997581049705-21554-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.12.189:46377
--webserver_interface=127.21.12.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.189.keytab
--principal=kudu/127.21.12.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.148:44185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.12.190:40733,127.21.12.189:46377,127.21.12.188:39245
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785997581049705-21554-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785997581049705-21554-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785997581049705-21554-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 06:28:53.408159 8914 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.12.189" port: 46377 } attempt: 1
I20260806 06:28:53.408676 8916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:28:53.408483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.189:46377 (local address 127.0.0.1:36782)
0806 06:28:53.408587 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0806 06:28:53.408637 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.189:46377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260806 06:28:53.408983 8916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:28:53.408903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.189:46377 (local address 127.0.0.1:36788)
0806 06:28:53.408939 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0806 06:28:53.408964 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.189:46377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260806 06:28:53.409080 8914 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.12.189:46377: Network error: Client connection negotiation failed: client connection to 127.21.12.189:46377: connect: Connection refused (error 111)
W20260806 06:28:53.451264 8922 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 06:28:53.451459 8922 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 06:28:53.451480 8922 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 06:28:53.452591 8922 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 06:28:53.452633 8922 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 06:28:53.452646 8922 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 06:28:53.452670 8922 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 06:28:53.452693 8922 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 06:28:53.454209 8922 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.148:44185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785997581049705-21554-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785997581049705-21554-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.21.12.190:40733,127.21.12.189:46377,127.21.12.188:39245
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.189.keytab
--principal=kudu/127.21.12.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.12.189:46377
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785997581049705-21554-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.21.12.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8922
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785997581049705-21554-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 05:11:02 UTC on ccycloud.quasar-hvzbbz.root.comops.site
build id 4569617
I20260806 06:28:53.454473 8922 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 06:28:53.454685 8922 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 06:28:53.459362 8930 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:28:53.459380 8933 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:28:53.459362 8935 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:28:53.459605 8922 server_base.cc:1034] running on GCE node
I20260806 06:28:53.459828 8922 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 06:28:53.460100 8922 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 06:28:53.461264 8922 hybrid_clock.cc:648] HybridClock initialized: now 1785997733461249 us; error 28 us; skew 500 ppm
Aug 06 06:28:53 dist-test-slave-4tdj krb5kdc[8794](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997733, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 06:28:53.463706 8922 init.cc:377] Logged in from keytab as kudu/127.21.12.189@KRBTEST.COM (short username kudu)
I20260806 06:28:53.464140 8922 webserver.cc:460] Webserver started at http://127.21.12.189:46709/ using document root <none> and password file <none>
I20260806 06:28:53.464331 8922 fs_manager.cc:359] Metadata directory not provided
I20260806 06:28:53.464378 8922 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 06:28:53.464505 8922 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 06:28:53.465539 8922 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785997581049705-21554-0/minicluster-data/master-1/data/instance:
uuid: "808c97503f2e438fbbf351cb34758102"
format_stamp: "Formatted at 2026-08-06 06:28:53 on dist-test-slave-4tdj"
server_key: "8ed4320f1aeedf4d8dbe7d11c4c70f6d"
server_key_iv: "06c3c5cc9c33a4f4761e29e9e8c8f330"
server_key_version: "encryptionkey@0"
I20260806 06:28:53.465874 8922 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785997581049705-21554-0/minicluster-data/master-1/wal/instance:
uuid: "808c97503f2e438fbbf351cb34758102"
format_stamp: "Formatted at 2026-08-06 06:28:53 on dist-test-slave-4tdj"
server_key: "8ed4320f1aeedf4d8dbe7d11c4c70f6d"
server_key_iv: "06c3c5cc9c33a4f4761e29e9e8c8f330"
server_key_version: "encryptionkey@0"
I20260806 06:28:53.467139 8922 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260806 06:28:53.467767 8950 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 06:28:53.467913 8922 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260806 06:28:53.468004 8922 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785997581049705-21554-0/minicluster-data/master-1/data,/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785997581049705-21554-0/minicluster-data/master-1/wal
uuid: "808c97503f2e438fbbf351cb34758102"
format_stamp: "Formatted at 2026-08-06 06:28:53 on dist-test-slave-4tdj"
server_key: "8ed4320f1aeedf4d8dbe7d11c4c70f6d"
server_key_iv: "06c3c5cc9c33a4f4761e29e9e8c8f330"
server_key_version: "encryptionkey@0"
I20260806 06:28:53.468060 8922 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785997581049705-21554-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785997581049705-21554-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785997581049705-21554-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 06:28:53.477218 8914 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.12.189" port: 46377 } attempt: 2
I20260806 06:28:53.477614 8916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:28:53.477472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.189:46377 (local address 127.0.0.1:36804)
0806 06:28:53.477534 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0806 06:28:53.477578 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.189:46377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260806 06:28:53.477911 8916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:28:53.477841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.189:46377 (local address 127.0.0.1:36818)
0806 06:28:53.477875 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0806 06:28:53.477896 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.189:46377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260806 06:28:53.477993 8914 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.12.189:46377: Network error: Client connection negotiation failed: client connection to 127.21.12.189:46377: connect: Connection refused (error 111)
I20260806 06:28:53.492971 8922 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 06:28:53.495077 8922 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 06:28:53.495198 8922 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 06:28:53.499047 8922 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.189:46377
I20260806 06:28:53.499084 9030 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.189:46377 every 8 connection(s)
I20260806 06:28:53.499396 8922 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785997581049705-21554-0/minicluster-data/master-1/data/info.pb
I20260806 06:28:53.500206 9032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 06:28:53.501659 9032 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.12.190" port: 40733 } has no permanent_uuid. Determining permanent_uuid...
I20260806 06:28:53.505571 21554 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHhLhT/build/release/bin/kudu as pid 8922
I20260806 06:28:53.505661 21554 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785997581049705-21554-0/minicluster-data/master-1/wal/instance
I20260806 06:28:53.505828 21554 external_mini_cluster.cc:1428] Setting key a4fe182530c4f567a794573beeed2547
Aug 06 06:28:53 dist-test-slave-4tdj krb5kdc[8794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997733, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.189@KRBTEST.COM for kudu/127.21.12.190@KRBTEST.COM
I20260806 06:28:53.509758 8917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:28:53.502320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54320 (local address 127.21.12.190:40733)
0806 06:28:53.502363 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0806 06:28:53.502365 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:28:53.502525 (+ 160us) server_negotiation.cc:407] Connection header received
0806 06:28:53.502968 (+ 443us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:28:53.502969 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:28:53.502980 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:28:53.503019 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:28:53.503585 (+ 566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:28:53.504058 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:28:53.504922 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:28:53.505104 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:28:53.505148 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:28:53.507210 (+ 2062us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:28:53.507212 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:28:53.507212 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 06:28:53.507219 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:28:53.508476 (+ 1257us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:28:53.508896 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:28:53.508897 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:28:53.508897 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:28:53.508954 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:28:53.509199 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:28:53.509200 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:28:53.509200 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:28:53.509294 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:28:53.509338 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0806 06:28:53.509624 (+ 286us) server_negotiation.cc:299] Negotiation successful
0806 06:28:53.509695 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260806 06:28:53.509977 9036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:28:53.502249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:40733 (local address 127.0.0.1:54320)
0806 06:28:53.502406 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0806 06:28:53.502447 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0806 06:28:53.502879 (+ 432us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:28:53.503086 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:28:53.503090 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:28:53.503319 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0806 06:28:53.503518 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:28:53.503522 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:28:53.504113 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:28:53.504114 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:28:53.504863 (+ 749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:28:53.504865 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:28:53.505188 (+ 323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:28:53.505189 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:28:53.505308 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:28:53.505786 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 06:28:53.505794 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0806 06:28:53.507116 (+ 1322us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 06:28:53.508555 (+ 1439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:28:53.508564 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:28:53.508566 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:28:53.508821 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:28:53.509016 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:28:53.509016 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:28:53.509017 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:28:53.509143 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:28:53.509344 (+ 201us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 06:28:53.509350 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 06:28:53.509507 (+ 157us) client_negotiation.cc:769] Sending connection context
0806 06:28:53.509605 (+ 98us) client_negotiation.cc:240] Negotiation successful
0806 06:28:53.509748 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":89,"threads_started":1}
I20260806 06:28:53.510535 9032 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.12.189" port: 46377 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.21.12.188@KRBTEST.COM; defaulting to no policy
Aug 06 06:28:53 dist-test-slave-4tdj krb5kdc[8794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997733, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.189@KRBTEST.COM for kudu/127.21.12.189@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.12.188@KRBTEST.COM" created.
I20260806 06:28:53.517339 9036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:28:53.510767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.189:46377 (local address 127.0.0.1:36834)
0806 06:28:53.510815 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0806 06:28:53.510827 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 06:28:53.510871 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:28:53.511078 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:28:53.511079 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:28:53.511212 (+ 133us) client_negotiation.cc:189] Negotiated authn=SASL
0806 06:28:53.511266 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:28:53.511268 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:28:53.512031 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:28:53.512032 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:28:53.512778 (+ 746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:28:53.512781 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:28:53.513174 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:28:53.513175 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:28:53.513249 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:28:53.513550 (+ 301us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 06:28:53.513555 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0806 06:28:53.514745 (+ 1190us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 06:28:53.516300 (+ 1555us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:28:53.516301 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:28:53.516302 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:28:53.516524 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:28:53.516690 (+ 166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:28:53.516690 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:28:53.516691 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:28:53.516743 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:28:53.516976 (+ 233us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 06:28:53.516978 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 06:28:53.517027 (+ 49us) client_negotiation.cc:769] Sending connection context
0806 06:28:53.517109 (+ 82us) client_negotiation.cc:240] Negotiation successful
0806 06:28:53.517212 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260806 06:28:53.517426 9039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:28:53.510811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36834 (local address 127.21.12.189:46377)
0806 06:28:53.510949 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0806 06:28:53.510954 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 06:28:53.510971 (+ 17us) server_negotiation.cc:407] Connection header received
0806 06:28:53.510992 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:28:53.510994 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:28:53.511005 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:28:53.511063 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:28:53.511330 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:28:53.511957 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:28:53.512847 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:28:53.513111 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:28:53.513155 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:28:53.514824 (+ 1669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:28:53.514832 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:28:53.514832 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 06:28:53.514838 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:28:53.516145 (+ 1307us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:28:53.516597 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:28:53.516599 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:28:53.516600 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:28:53.516644 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:28:53.516790 (+ 146us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:28:53.516790 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:28:53.516790 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:28:53.516901 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:28:53.516956 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0806 06:28:53.517138 (+ 182us) server_negotiation.cc:299] Negotiation successful
0806 06:28:53.517270 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":60,"server-negotiator.queue_time_us":110,"thread_start_us":75,"threads_started":1}
I20260806 06:28:53.518901 9032 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.12.188" port: 39245 } has no permanent_uuid. Determining permanent_uuid...
I20260806 06:28:53.519207 9036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:28:53.519096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.188:39245 (local address 127.0.0.1:41504)
0806 06:28:53.519136 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0806 06:28:53.519183 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.188:39245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260806 06:28:53.519325 8954 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.12.188:39245: connect: Connection refused (error 111)
I20260806 06:28:53.519661 9036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:28:53.519582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.188:39245 (local address 127.0.0.1:41516)
0806 06:28:53.519617 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0806 06:28:53.519640 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.188:39245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260806 06:28:53.519793 9032 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.12.188:39245: Network error: Client connection negotiation failed: client connection to 127.21.12.188:39245: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.12.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.188.keytab.
Entry for principal kudu/127.21.12.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.188.keytab.
I20260806 06:28:53.530727 21554 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHhLhT/build/release/bin/kudu
/tmp/dist-test-taskFHhLhT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785997581049705-21554-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785997581049705-21554-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785997581049705-21554-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785997581049705-21554-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.12.188:39245
--webserver_interface=127.21.12.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.188.keytab
--principal=kudu/127.21.12.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.148:44185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.12.190:40733,127.21.12.189:46377,127.21.12.188:39245
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785997581049705-21554-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785997581049705-21554-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785997581049705-21554-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 06:28:53.569126 8914 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.12.189" port: 46377 } attempt: 3
I20260806 06:28:53.569080 9032 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.12.188" port: 39245 } attempt: 1
I20260806 06:28:53.569553 9036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:28:53.569424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.188:39245 (local address 127.0.0.1:41520)
0806 06:28:53.569486 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0806 06:28:53.569519 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.188:39245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260806 06:28:53.569900 9036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:28:53.569814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.188:39245 (local address 127.0.0.1:41522)
0806 06:28:53.569843 (+ 29us) negotiation.cc:107] Waiting for socket to connect
0806 06:28:53.569873 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.188:39245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
W20260806 06:28:53.569988 9032 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.12.188:39245: Network error: Client connection negotiation failed: client connection to 127.21.12.188:39245: connect: Connection refused (error 111)
Aug 06 06:28:53 dist-test-slave-4tdj krb5kdc[8794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997733, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.190@KRBTEST.COM for kudu/127.21.12.189@KRBTEST.COM
I20260806 06:28:53.575595 9039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:28:53.569493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36836 (local address 127.21.12.189:46377)
0806 06:28:53.569529 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0806 06:28:53.569531 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:28:53.569543 (+ 12us) server_negotiation.cc:407] Connection header received
0806 06:28:53.569707 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:28:53.569708 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:28:53.569719 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:28:53.569774 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:28:53.570161 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:28:53.570747 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:28:53.571439 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:28:53.571641 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:28:53.571726 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:28:53.573395 (+ 1669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:28:53.573396 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:28:53.573397 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 06:28:53.573404 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:28:53.574649 (+ 1245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:28:53.574933 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:28:53.574934 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:28:53.574934 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:28:53.574984 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:28:53.575140 (+ 156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:28:53.575141 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:28:53.575142 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:28:53.575254 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:28:53.575309 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0806 06:28:53.575442 (+ 133us) server_negotiation.cc:299] Negotiation successful
0806 06:28:53.575510 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260806 06:28:53.575600 8916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:28:53.569447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.189:46377 (local address 127.0.0.1:36836)
0806 06:28:53.569500 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0806 06:28:53.569511 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0806 06:28:53.569556 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:28:53.569787 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:28:53.569789 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:28:53.570005 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0806 06:28:53.570086 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:28:53.570089 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:28:53.570825 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:28:53.570827 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:28:53.571363 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:28:53.571365 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:28:53.571738 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:28:53.571739 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:28:53.571814 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:28:53.572209 (+ 395us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 06:28:53.572216 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0806 06:28:53.573321 (+ 1105us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 06:28:53.574720 (+ 1399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:28:53.574721 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:28:53.574722 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:28:53.574875 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:28:53.575035 (+ 160us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:28:53.575036 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:28:53.575036 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:28:53.575091 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:28:53.575309 (+ 218us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 06:28:53.575310 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 06:28:53.575354 (+ 44us) client_negotiation.cc:769] Sending connection context
0806 06:28:53.575413 (+ 59us) client_negotiation.cc:240] Negotiation successful
0806 06:28:53.575506 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260806 06:28:53.576134 8914 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.12.188" port: 39245 } has no permanent_uuid. Determining permanent_uuid...
I20260806 06:28:53.576449 8916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:28:53.576366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.188:39245 (local address 127.0.0.1:41532)
0806 06:28:53.576406 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0806 06:28:53.576433 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.188:39245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260806 06:28:53.576740 8916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:28:53.576656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.188:39245 (local address 127.0.0.1:41534)
0806 06:28:53.576696 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0806 06:28:53.576719 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.188:39245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260806 06:28:53.576821 8914 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.12.188:39245: Network error: Client connection negotiation failed: client connection to 127.21.12.188:39245: connect: Connection refused (error 111)
I20260806 06:28:53.607954 8914 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.12.188" port: 39245 } attempt: 1
I20260806 06:28:53.608343 8916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:28:53.608237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.188:39245 (local address 127.0.0.1:41538)
0806 06:28:53.608290 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0806 06:28:53.608318 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.188:39245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260806 06:28:53.608639 8916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:28:53.608562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.188:39245 (local address 127.0.0.1:41542)
0806 06:28:53.608600 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0806 06:28:53.608622 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.188:39245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260806 06:28:53.608727 8914 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.12.188:39245: Network error: Client connection negotiation failed: client connection to 127.21.12.188:39245: connect: Connection refused (error 111)
W20260806 06:28:53.611465 9044 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 06:28:53.611663 9044 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 06:28:53.611694 9044 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 06:28:53.612778 9044 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 06:28:53.612815 9044 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 06:28:53.612834 9044 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 06:28:53.612849 9044 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 06:28:53.612866 9044 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 06:28:53.614384 9044 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.148:44185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785997581049705-21554-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785997581049705-21554-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.21.12.190:40733,127.21.12.189:46377,127.21.12.188:39245
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.188.keytab
--principal=kudu/127.21.12.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.12.188:39245
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785997581049705-21554-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.21.12.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9044
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785997581049705-21554-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 05:11:02 UTC on ccycloud.quasar-hvzbbz.root.comops.site
build id 4569617
I20260806 06:28:53.614626 9044 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 06:28:53.614820 9044 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 06:28:53.619382 9052 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:28:53.619391 9054 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:28:53.619397 9044 server_base.cc:1034] running on GCE node
W20260806 06:28:53.619382 9056 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:28:53.619773 9044 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 06:28:53.620026 9044 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 06:28:53.621168 9044 hybrid_clock.cc:648] HybridClock initialized: now 1785997733621156 us; error 28 us; skew 500 ppm
Aug 06 06:28:53 dist-test-slave-4tdj krb5kdc[8794](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997733, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 06:28:53.623505 9044 init.cc:377] Logged in from keytab as kudu/127.21.12.188@KRBTEST.COM (short username kudu)
I20260806 06:28:53.623953 9044 webserver.cc:460] Webserver started at http://127.21.12.188:43325/ using document root <none> and password file <none>
I20260806 06:28:53.624142 9044 fs_manager.cc:359] Metadata directory not provided
I20260806 06:28:53.624188 9044 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 06:28:53.624310 9044 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 06:28:53.625294 9044 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785997581049705-21554-0/minicluster-data/master-2/data/instance:
uuid: "a610668bf3dc4846a8f809bc0f652443"
format_stamp: "Formatted at 2026-08-06 06:28:53 on dist-test-slave-4tdj"
server_key: "77c18b95e64ebc587fb1f8a9510d4150"
server_key_iv: "229d83271d3bcc850bc2cc9f7e7348f5"
server_key_version: "encryptionkey@0"
I20260806 06:28:53.625620 9044 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785997581049705-21554-0/minicluster-data/master-2/wal/instance:
uuid: "a610668bf3dc4846a8f809bc0f652443"
format_stamp: "Formatted at 2026-08-06 06:28:53 on dist-test-slave-4tdj"
server_key: "77c18b95e64ebc587fb1f8a9510d4150"
server_key_iv: "229d83271d3bcc850bc2cc9f7e7348f5"
server_key_version: "encryptionkey@0"
I20260806 06:28:53.626849 9044 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260806 06:28:53.627465 9065 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 06:28:53.627627 9044 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260806 06:28:53.627709 9044 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785997581049705-21554-0/minicluster-data/master-2/data,/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785997581049705-21554-0/minicluster-data/master-2/wal
uuid: "a610668bf3dc4846a8f809bc0f652443"
format_stamp: "Formatted at 2026-08-06 06:28:53 on dist-test-slave-4tdj"
server_key: "77c18b95e64ebc587fb1f8a9510d4150"
server_key_iv: "229d83271d3bcc850bc2cc9f7e7348f5"
server_key_version: "encryptionkey@0"
I20260806 06:28:53.627766 9044 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785997581049705-21554-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785997581049705-21554-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785997581049705-21554-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 06:28:53.637897 9044 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 06:28:53.638139 9032 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.12.188" port: 39245 } attempt: 2
I20260806 06:28:53.638512 9036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:28:53.638397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.188:39245 (local address 127.0.0.1:41550)
0806 06:28:53.638445 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0806 06:28:53.638480 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.188:39245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260806 06:28:53.638854 9036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:28:53.638756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.188:39245 (local address 127.0.0.1:41566)
0806 06:28:53.638802 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0806 06:28:53.638828 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.188:39245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260806 06:28:53.638962 9032 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.12.188:39245: Network error: Client connection negotiation failed: client connection to 127.21.12.188:39245: connect: Connection refused (error 111)
I20260806 06:28:53.639915 9044 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 06:28:53.640043 9044 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 06:28:53.643652 9044 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.188:39245
I20260806 06:28:53.643683 9141 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.188:39245 every 8 connection(s)
I20260806 06:28:53.643987 9044 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785997581049705-21554-0/minicluster-data/master-2/data/info.pb
I20260806 06:28:53.644800 9142 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 06:28:53.645130 21554 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHhLhT/build/release/bin/kudu as pid 9044
I20260806 06:28:53.645236 21554 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785997581049705-21554-0/minicluster-data/master-2/wal/instance
I20260806 06:28:53.645401 21554 external_mini_cluster.cc:1428] Setting key 5deba1bfcc649672559bd2837b276b7a
I20260806 06:28:53.646515 9142 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.12.190" port: 40733 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 06:28:53 dist-test-slave-4tdj krb5kdc[8794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997733, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.188@KRBTEST.COM for kudu/127.21.12.190@KRBTEST.COM
Aug 06 06:28:53 dist-test-slave-4tdj krb5kdc[8794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997733, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.12.188@KRBTEST.COM
I20260806 06:28:53.655197 8917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:28:53.647104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54328 (local address 127.21.12.190:40733)
0806 06:28:53.647153 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0806 06:28:53.647155 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:28:53.647281 (+ 126us) server_negotiation.cc:407] Connection header received
0806 06:28:53.647340 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:28:53.647341 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:28:53.647350 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:28:53.647417 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:28:53.647820 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:28:53.648526 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:28:53.649686 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:28:53.649973 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:28:53.650044 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:28:53.652182 (+ 2138us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:28:53.652183 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:28:53.652184 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 06:28:53.652191 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:28:53.653727 (+ 1536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:28:53.654231 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:28:53.654232 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:28:53.654233 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:28:53.654314 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:28:53.654560 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:28:53.654561 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:28:53.654561 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:28:53.654692 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:28:53.654748 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0806 06:28:53.655010 (+ 262us) server_negotiation.cc:299] Negotiation successful
0806 06:28:53.655094 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260806 06:28:53.655380 9148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:28:53.647020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:40733 (local address 127.0.0.1:54328)
0806 06:28:53.647165 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0806 06:28:53.647217 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0806 06:28:53.647289 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:28:53.647431 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:28:53.647433 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:28:53.647648 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0806 06:28:53.647743 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:28:53.647747 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:28:53.648608 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:28:53.648608 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:28:53.649621 (+ 1013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:28:53.649624 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:28:53.650041 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:28:53.650041 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:28:53.650110 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:28:53.650561 (+ 451us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 06:28:53.650569 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0806 06:28:53.652100 (+ 1531us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 06:28:53.653816 (+ 1716us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:28:53.653823 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:28:53.653824 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:28:53.654161 (+ 337us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:28:53.654363 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:28:53.654365 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:28:53.654365 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:28:53.654491 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:28:53.654751 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 06:28:53.654758 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 06:28:53.654887 (+ 129us) client_negotiation.cc:769] Sending connection context
0806 06:28:53.654974 (+ 87us) client_negotiation.cc:240] Negotiation successful
0806 06:28:53.655134 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":77,"threads_started":1}
I20260806 06:28:53.655383 9146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:28:53.645852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41578 (local address 127.21.12.188:39245)
0806 06:28:53.646044 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0806 06:28:53.646052 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 06:28:53.646075 (+ 23us) server_negotiation.cc:407] Connection header received
0806 06:28:53.646505 (+ 430us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:28:53.646511 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:28:53.646557 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:28:53.646683 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:28:53.647339 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:28:53.648540 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:28:53.649560 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:28:53.649962 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:28:53.650065 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:28:53.652626 (+ 2561us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:28:53.652634 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:28:53.652634 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 06:28:53.652641 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:28:53.654178 (+ 1537us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:28:53.654567 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:28:53.654570 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:28:53.654570 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:28:53.654621 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:28:53.654794 (+ 173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:28:53.654794 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:28:53.654795 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:28:53.654919 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:28:53.655036 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0806 06:28:53.655119 (+ 83us) server_negotiation.cc:299] Negotiation successful
0806 06:28:53.655186 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":86,"threads_started":1}
I20260806 06:28:53.656019 9142 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.12.189" port: 46377 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 06:28:53 dist-test-slave-4tdj krb5kdc[8794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997733, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.188@KRBTEST.COM for kudu/127.21.12.189@KRBTEST.COM
Aug 06 06:28:53 dist-test-slave-4tdj krb5kdc[8794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997733, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.12.190@KRBTEST.COM
I20260806 06:28:53.661638 9039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:28:53.656319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36840 (local address 127.21.12.189:46377)
0806 06:28:53.656367 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0806 06:28:53.656369 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:28:53.656376 (+ 7us) server_negotiation.cc:407] Connection header received
0806 06:28:53.656401 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:28:53.656402 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:28:53.656411 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:28:53.656455 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:28:53.656664 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:28:53.657108 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:28:53.657697 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:28:53.657903 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:28:53.657955 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:28:53.659536 (+ 1581us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:28:53.659538 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:28:53.659538 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 06:28:53.659545 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:28:53.660718 (+ 1173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:28:53.660995 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:28:53.660996 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:28:53.660997 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:28:53.661049 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:28:53.661208 (+ 159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:28:53.661209 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:28:53.661209 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:28:53.661312 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:28:53.661364 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0806 06:28:53.661489 (+ 125us) server_negotiation.cc:299] Negotiation successful
0806 06:28:53.661553 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260806 06:28:53.661634 9148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:28:53.656227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.189:46377 (local address 127.0.0.1:36840)
0806 06:28:53.656309 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0806 06:28:53.656319 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0806 06:28:53.656358 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:28:53.656456 (+ 98us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:28:53.656457 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:28:53.656578 (+ 121us) client_negotiation.cc:189] Negotiated authn=SASL
0806 06:28:53.656604 (+ 26us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:28:53.656605 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:28:53.657170 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:28:53.657171 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:28:53.657631 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:28:53.657633 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:28:53.657957 (+ 324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:28:53.657958 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:28:53.658010 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:28:53.658356 (+ 346us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 06:28:53.658360 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0806 06:28:53.659462 (+ 1102us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 06:28:53.660788 (+ 1326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:28:53.660789 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:28:53.660789 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:28:53.660938 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:28:53.661101 (+ 163us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:28:53.661101 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:28:53.661102 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:28:53.661153 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:28:53.661363 (+ 210us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 06:28:53.661364 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 06:28:53.661402 (+ 38us) client_negotiation.cc:769] Sending connection context
0806 06:28:53.661462 (+ 60us) client_negotiation.cc:240] Negotiation successful
0806 06:28:53.661545 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69}
I20260806 06:28:53.662277 8917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:28:53.656276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54338 (local address 127.21.12.190:40733)
0806 06:28:53.656312 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0806 06:28:53.656314 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:28:53.656323 (+ 9us) server_negotiation.cc:407] Connection header received
0806 06:28:53.656404 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:28:53.656405 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:28:53.656415 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:28:53.656460 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:28:53.656875 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:28:53.657304 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:28:53.657914 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:28:53.658096 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:28:53.658149 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:28:53.660080 (+ 1931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:28:53.660081 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:28:53.660082 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 06:28:53.660088 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:28:53.661288 (+ 1200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:28:53.661573 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:28:53.661574 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:28:53.661574 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:28:53.661621 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:28:53.661780 (+ 159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:28:53.661781 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:28:53.661781 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:28:53.661875 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:28:53.662035 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0806 06:28:53.662079 (+ 44us) server_negotiation.cc:299] Negotiation successful
0806 06:28:53.662142 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260806 06:28:53.662338 9142 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.12.188" port: 39245 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 06:28:53 dist-test-slave-4tdj krb5kdc[8794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997733, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.188@KRBTEST.COM for kudu/127.21.12.188@KRBTEST.COM
Aug 06 06:28:53 dist-test-slave-4tdj krb5kdc[8794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997733, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.12.189@KRBTEST.COM
I20260806 06:28:53.667936 9148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:28:53.662552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.188:39245 (local address 127.0.0.1:41582)
0806 06:28:53.662587 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0806 06:28:53.662596 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0806 06:28:53.662625 (+ 29us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:28:53.662736 (+ 111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:28:53.662736 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:28:53.662853 (+ 117us) client_negotiation.cc:189] Negotiated authn=SASL
0806 06:28:53.662879 (+ 26us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:28:53.662880 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:28:53.663513 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:28:53.663514 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:28:53.663957 (+ 443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:28:53.663959 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:28:53.664283 (+ 324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:28:53.664284 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:28:53.664327 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:28:53.664655 (+ 328us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 06:28:53.664658 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0806 06:28:53.665635 (+ 977us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 06:28:53.667040 (+ 1405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:28:53.667041 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:28:53.667041 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:28:53.667198 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:28:53.667359 (+ 161us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:28:53.667360 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:28:53.667360 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:28:53.667417 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:28:53.667648 (+ 231us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 06:28:53.667649 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 06:28:53.667680 (+ 31us) client_negotiation.cc:769] Sending connection context
0806 06:28:53.667737 (+ 57us) client_negotiation.cc:240] Negotiation successful
0806 06:28:53.667821 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24}
I20260806 06:28:53.668040 9146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:28:53.662620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41582 (local address 127.21.12.188:39245)
0806 06:28:53.662653 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0806 06:28:53.662655 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:28:53.662661 (+ 6us) server_negotiation.cc:407] Connection header received
0806 06:28:53.662684 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:28:53.662685 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:28:53.662692 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:28:53.662734 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:28:53.662955 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:28:53.663450 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:28:53.664020 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:28:53.664229 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:28:53.664279 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:28:53.665705 (+ 1426us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:28:53.665706 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:28:53.665707 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 06:28:53.665713 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:28:53.666977 (+ 1264us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:28:53.667255 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:28:53.667256 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:28:53.667256 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:28:53.667305 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:28:53.667470 (+ 165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:28:53.667471 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:28:53.667471 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:28:53.667572 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:28:53.667627 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0806 06:28:53.667776 (+ 149us) server_negotiation.cc:299] Negotiation successful
0806 06:28:53.667870 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":31,"server-negotiator.queue_time_us":20}
I20260806 06:28:53.669615 9039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:28:53.663172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36846 (local address 127.21.12.189:46377)
0806 06:28:53.663219 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0806 06:28:53.663221 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:28:53.663229 (+ 8us) server_negotiation.cc:407] Connection header received
0806 06:28:53.663246 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:28:53.663247 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:28:53.663254 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:28:53.663296 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:28:53.663704 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:28:53.664127 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:28:53.664866 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:28:53.665058 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:28:53.665108 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:28:53.667310 (+ 2202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:28:53.667311 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:28:53.667311 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 06:28:53.667317 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:28:53.668688 (+ 1371us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:28:53.669049 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:28:53.669050 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:28:53.669050 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:28:53.669089 (+ 39us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:28:53.669243 (+ 154us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:28:53.669243 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:28:53.669244 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:28:53.669322 (+ 78us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:28:53.669438 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0806 06:28:53.669508 (+ 70us) server_negotiation.cc:299] Negotiation successful
0806 06:28:53.669559 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260806 06:28:53.669626 9142 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a610668bf3dc4846a8f809bc0f652443: Bootstrap starting.
I20260806 06:28:53.670837 9142 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a610668bf3dc4846a8f809bc0f652443: Neither blocks nor log segments found. Creating new log.
I20260806 06:28:53.670907 21554 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 06:28:53.671141 9142 log.cc:824] T 00000000000000000000000000000000 P a610668bf3dc4846a8f809bc0f652443: Log is configured to *not* fsync() on all Append() calls
I20260806 06:28:53.671947 9142 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a610668bf3dc4846a8f809bc0f652443: No bootstrap required, opened a new log
I20260806 06:28:53.673995 9142 raft_consensus.cc:348] T 00000000000000000000000000000000 P a610668bf3dc4846a8f809bc0f652443 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81239de6d9b14d648ebebd041840a733" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 40733 } } peers { permanent_uuid: "808c97503f2e438fbbf351cb34758102" member_type: VOTER last_known_addr { host: "127.21.12.189" port: 46377 } } peers { permanent_uuid: "a610668bf3dc4846a8f809bc0f652443" member_type: VOTER last_known_addr { host: "127.21.12.188" port: 39245 } }
I20260806 06:28:53.674156 9142 raft_consensus.cc:374] T 00000000000000000000000000000000 P a610668bf3dc4846a8f809bc0f652443 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 06:28:53.674201 9142 raft_consensus.cc:729] T 00000000000000000000000000000000 P a610668bf3dc4846a8f809bc0f652443 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a610668bf3dc4846a8f809bc0f652443, State: Initialized, Role: FOLLOWER
I20260806 06:28:53.674340 9142 consensus_queue.cc:260] T 00000000000000000000000000000000 P a610668bf3dc4846a8f809bc0f652443 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81239de6d9b14d648ebebd041840a733" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 40733 } } peers { permanent_uuid: "808c97503f2e438fbbf351cb34758102" member_type: VOTER last_known_addr { host: "127.21.12.189" port: 46377 } } peers { permanent_uuid: "a610668bf3dc4846a8f809bc0f652443" member_type: VOTER last_known_addr { host: "127.21.12.188" port: 39245 } }
I20260806 06:28:53.674654 9142 sys_catalog.cc:564] T 00000000000000000000000000000000 P a610668bf3dc4846a8f809bc0f652443 [sys.catalog]: configured and running, proceeding with master startup.
W20260806 06:28:53.678181 9176 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a610668bf3dc4846a8f809bc0f652443: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 06:28:53.678233 9176 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 06:28:53.678414 9161 sys_catalog.cc:455] T 00000000000000000000000000000000 P a610668bf3dc4846a8f809bc0f652443 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81239de6d9b14d648ebebd041840a733" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 40733 } } peers { permanent_uuid: "808c97503f2e438fbbf351cb34758102" member_type: VOTER last_known_addr { host: "127.21.12.189" port: 46377 } } peers { permanent_uuid: "a610668bf3dc4846a8f809bc0f652443" member_type: VOTER last_known_addr { host: "127.21.12.188" port: 39245 } } }
I20260806 06:28:53.678505 9161 sys_catalog.cc:458] T 00000000000000000000000000000000 P a610668bf3dc4846a8f809bc0f652443 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 06:28:53.679848 9146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:28:53.672490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41596 (local address 127.21.12.188:39245)
0806 06:28:53.672529 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0806 06:28:53.672530 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0806 06:28:53.672614 (+ 84us) server_negotiation.cc:407] Connection header received
0806 06:28:53.672669 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:28:53.672670 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:28:53.672686 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:28:53.672734 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:28:53.673377 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:28:53.674011 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:28:53.675083 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:28:53.675352 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:28:53.675392 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:28:53.676919 (+ 1527us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:28:53.676920 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:28:53.676921 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 06:28:53.676927 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:28:53.678687 (+ 1760us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:28:53.679048 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:28:53.679049 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:28:53.679049 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:28:53.679105 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:28:53.679289 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:28:53.679289 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:28:53.679290 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:28:53.679426 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:28:53.679489 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0806 06:28:53.679699 (+ 210us) server_negotiation.cc:299] Negotiation successful
0806 06:28:53.679772 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260806 06:28:53.680189 9039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:28:53.672323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36860 (local address 127.21.12.189:46377)
0806 06:28:53.672356 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0806 06:28:53.672358 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:28:53.673289 (+ 931us) server_negotiation.cc:407] Connection header received
0806 06:28:53.673369 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:28:53.673370 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:28:53.673379 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:28:53.673404 (+ 25us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:28:53.674018 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:28:53.674701 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:28:53.675621 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:28:53.675858 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:28:53.675889 (+ 31us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:28:53.677151 (+ 1262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:28:53.677153 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:28:53.677153 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 06:28:53.677159 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:28:53.678556 (+ 1397us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:28:53.678918 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:28:53.678919 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:28:53.678919 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:28:53.678975 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:28:53.679167 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:28:53.679168 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:28:53.679169 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:28:53.679297 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:28:53.679810 (+ 513us) server_negotiation.cc:1035] Waiting for connection context
0806 06:28:53.680024 (+ 214us) server_negotiation.cc:299] Negotiation successful
0806 06:28:53.680095 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260806 06:28:53.677501 9044 master_runner.cc:186] Error getting master registration for 127.21.12.190:40733: OK, instance_id { permanent_uuid: "81239de6d9b14d648ebebd041840a733" instance_seqno: 1785997733344238 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260806 06:28:53.680545 9044 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260806 06:28:53.682513 8917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:28:53.672453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54350 (local address 127.21.12.190:40733)
0806 06:28:53.672487 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0806 06:28:53.672488 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0806 06:28:53.672585 (+ 97us) server_negotiation.cc:407] Connection header received
0806 06:28:53.672669 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:28:53.672670 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:28:53.672679 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:28:53.672706 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:28:53.673206 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:28:53.673898 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:28:53.675130 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:28:53.675415 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:28:53.675451 (+ 36us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:28:53.677235 (+ 1784us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:28:53.677236 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:28:53.677236 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 06:28:53.677243 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:28:53.681560 (+ 4317us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:28:53.681844 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:28:53.681845 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:28:53.681845 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:28:53.681898 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:28:53.682051 (+ 153us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:28:53.682052 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:28:53.682052 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:28:53.682153 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:28:53.682236 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0806 06:28:53.682363 (+ 127us) server_negotiation.cc:299] Negotiation successful
0806 06:28:53.682431 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260806 06:28:53.683851 8914 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.12.188" port: 39245 } attempt: 2
Aug 06 06:28:53 dist-test-slave-4tdj krb5kdc[8794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997733, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.190@KRBTEST.COM for kudu/127.21.12.188@KRBTEST.COM
I20260806 06:28:53.689831 8916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:28:53.684054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.188:39245 (local address 127.0.0.1:41612)
0806 06:28:53.684200 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0806 06:28:53.684210 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0806 06:28:53.684243 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:28:53.684468 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:28:53.684470 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:28:53.684594 (+ 124us) client_negotiation.cc:189] Negotiated authn=SASL
0806 06:28:53.684625 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:28:53.684626 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:28:53.685175 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:28:53.685176 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:28:53.685649 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:28:53.685650 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:28:53.685963 (+ 313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:28:53.685964 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:28:53.686013 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:28:53.686438 (+ 425us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 06:28:53.686441 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0806 06:28:53.687564 (+ 1123us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 06:28:53.689035 (+ 1471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:28:53.689035 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:28:53.689036 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:28:53.689192 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:28:53.689364 (+ 172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:28:53.689365 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:28:53.689365 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:28:53.689406 (+ 41us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:28:53.689633 (+ 227us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 06:28:53.689633 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 06:28:53.689664 (+ 31us) client_negotiation.cc:769] Sending connection context
0806 06:28:53.689708 (+ 44us) client_negotiation.cc:240] Negotiation successful
0806 06:28:53.689769 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130}
I20260806 06:28:53.690186 9146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:28:53.684304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41612 (local address 127.21.12.188:39245)
0806 06:28:53.684383 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0806 06:28:53.684385 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:28:53.684392 (+ 7us) server_negotiation.cc:407] Connection header received
0806 06:28:53.684408 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:28:53.684409 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:28:53.684417 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:28:53.684456 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:28:53.684683 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:28:53.685112 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:28:53.685720 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:28:53.685909 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:28:53.685959 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:28:53.687649 (+ 1690us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:28:53.687650 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:28:53.687651 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 06:28:53.687657 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:28:53.688964 (+ 1307us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:28:53.689253 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:28:53.689253 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:28:53.689254 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:28:53.689308 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:28:53.689453 (+ 145us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:28:53.689454 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:28:53.689454 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:28:53.689581 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:28:53.689643 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0806 06:28:53.689731 (+ 88us) server_negotiation.cc:299] Negotiation successful
0806 06:28:53.689798 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260806 06:28:53.691414 8914 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81239de6d9b14d648ebebd041840a733: Bootstrap starting.
I20260806 06:28:53.692037 8914 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 81239de6d9b14d648ebebd041840a733: Neither blocks nor log segments found. Creating new log.
I20260806 06:28:53.692351 8914 log.cc:824] T 00000000000000000000000000000000 P 81239de6d9b14d648ebebd041840a733: Log is configured to *not* fsync() on all Append() calls
I20260806 06:28:53.693066 8914 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81239de6d9b14d648ebebd041840a733: No bootstrap required, opened a new log
I20260806 06:28:53.694499 8914 raft_consensus.cc:348] T 00000000000000000000000000000000 P 81239de6d9b14d648ebebd041840a733 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81239de6d9b14d648ebebd041840a733" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 40733 } } peers { permanent_uuid: "808c97503f2e438fbbf351cb34758102" member_type: VOTER last_known_addr { host: "127.21.12.189" port: 46377 } } peers { permanent_uuid: "a610668bf3dc4846a8f809bc0f652443" member_type: VOTER last_known_addr { host: "127.21.12.188" port: 39245 } }
I20260806 06:28:53.694605 8914 raft_consensus.cc:374] T 00000000000000000000000000000000 P 81239de6d9b14d648ebebd041840a733 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 06:28:53.694639 8914 raft_consensus.cc:729] T 00000000000000000000000000000000 P 81239de6d9b14d648ebebd041840a733 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81239de6d9b14d648ebebd041840a733, State: Initialized, Role: FOLLOWER
I20260806 06:28:53.694758 8914 consensus_queue.cc:260] T 00000000000000000000000000000000 P 81239de6d9b14d648ebebd041840a733 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81239de6d9b14d648ebebd041840a733" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 40733 } } peers { permanent_uuid: "808c97503f2e438fbbf351cb34758102" member_type: VOTER last_known_addr { host: "127.21.12.189" port: 46377 } } peers { permanent_uuid: "a610668bf3dc4846a8f809bc0f652443" member_type: VOTER last_known_addr { host: "127.21.12.188" port: 39245 } }
I20260806 06:28:53.695045 8914 sys_catalog.cc:564] T 00000000000000000000000000000000 P 81239de6d9b14d648ebebd041840a733 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 06:28:53.695009 9180 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81239de6d9b14d648ebebd041840a733 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81239de6d9b14d648ebebd041840a733" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 40733 } } peers { permanent_uuid: "808c97503f2e438fbbf351cb34758102" member_type: VOTER last_known_addr { host: "127.21.12.189" port: 46377 } } peers { permanent_uuid: "a610668bf3dc4846a8f809bc0f652443" member_type: VOTER last_known_addr { host: "127.21.12.188" port: 39245 } } }
I20260806 06:28:53.695103 9180 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81239de6d9b14d648ebebd041840a733 [sys.catalog]: This master's current role is: FOLLOWER
W20260806 06:28:53.696673 9202 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 81239de6d9b14d648ebebd041840a733: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 06:28:53.696729 9202 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 06:28:53.697521 8816 master_runner.cc:186] Error getting master registration for 127.21.12.189:46377: OK, instance_id { permanent_uuid: "808c97503f2e438fbbf351cb34758102" instance_seqno: 1785997733497951 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260806 06:28:53.700362 8816 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260806 06:28:53.730114 9032 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.12.188" port: 39245 } attempt: 3
Aug 06 06:28:53 dist-test-slave-4tdj krb5kdc[8794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997733, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.189@KRBTEST.COM for kudu/127.21.12.188@KRBTEST.COM
I20260806 06:28:53.738144 9146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:28:53.730551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41624 (local address 127.21.12.188:39245)
0806 06:28:53.730639 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0806 06:28:53.730641 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:28:53.730653 (+ 12us) server_negotiation.cc:407] Connection header received
0806 06:28:53.730675 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:28:53.730676 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:28:53.730685 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:28:53.730738 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:28:53.731106 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:28:53.731860 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:28:53.732879 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:28:53.733174 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:28:53.733517 (+ 343us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:28:53.735473 (+ 1956us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:28:53.735475 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:28:53.735476 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 06:28:53.735484 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:28:53.737016 (+ 1532us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:28:53.737391 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:28:53.737392 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:28:53.737392 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:28:53.737454 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:28:53.737641 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:28:53.737642 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:28:53.737642 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:28:53.737768 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:28:53.737835 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0806 06:28:53.737978 (+ 143us) server_negotiation.cc:299] Negotiation successful
0806 06:28:53.738053 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260806 06:28:53.738155 9036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:28:53.730449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.188:39245 (local address 127.0.0.1:41624)
0806 06:28:53.730529 (+ 80us) negotiation.cc:107] Waiting for socket to connect
0806 06:28:53.730541 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 06:28:53.730594 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:28:53.730761 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:28:53.730766 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:28:53.730971 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0806 06:28:53.731026 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:28:53.731029 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:28:53.731957 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:28:53.731958 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:28:53.732748 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:28:53.732752 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:28:53.733279 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:28:53.733281 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:28:53.733340 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:28:53.733845 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 06:28:53.733851 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0806 06:28:53.735377 (+ 1526us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 06:28:53.737099 (+ 1722us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:28:53.737101 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:28:53.737101 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:28:53.737318 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:28:53.737518 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:28:53.737519 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:28:53.737519 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:28:53.737582 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:28:53.737831 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 06:28:53.737832 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 06:28:53.737879 (+ 47us) client_negotiation.cc:769] Sending connection context
0806 06:28:53.737946 (+ 67us) client_negotiation.cc:240] Negotiation successful
0806 06:28:53.738051 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260806 06:28:53.740183 9032 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 808c97503f2e438fbbf351cb34758102: Bootstrap starting.
I20260806 06:28:53.740921 9032 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 808c97503f2e438fbbf351cb34758102: Neither blocks nor log segments found. Creating new log.
I20260806 06:28:53.741233 9032 log.cc:824] T 00000000000000000000000000000000 P 808c97503f2e438fbbf351cb34758102: Log is configured to *not* fsync() on all Append() calls
I20260806 06:28:53.742149 9032 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 808c97503f2e438fbbf351cb34758102: No bootstrap required, opened a new log
I20260806 06:28:53.744621 9032 raft_consensus.cc:348] T 00000000000000000000000000000000 P 808c97503f2e438fbbf351cb34758102 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81239de6d9b14d648ebebd041840a733" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 40733 } } peers { permanent_uuid: "808c97503f2e438fbbf351cb34758102" member_type: VOTER last_known_addr { host: "127.21.12.189" port: 46377 } } peers { permanent_uuid: "a610668bf3dc4846a8f809bc0f652443" member_type: VOTER last_known_addr { host: "127.21.12.188" port: 39245 } }
I20260806 06:28:53.744762 9032 raft_consensus.cc:374] T 00000000000000000000000000000000 P 808c97503f2e438fbbf351cb34758102 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 06:28:53.744796 9032 raft_consensus.cc:729] T 00000000000000000000000000000000 P 808c97503f2e438fbbf351cb34758102 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 808c97503f2e438fbbf351cb34758102, State: Initialized, Role: FOLLOWER
I20260806 06:28:53.744930 9032 consensus_queue.cc:260] T 00000000000000000000000000000000 P 808c97503f2e438fbbf351cb34758102 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81239de6d9b14d648ebebd041840a733" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 40733 } } peers { permanent_uuid: "808c97503f2e438fbbf351cb34758102" member_type: VOTER last_known_addr { host: "127.21.12.189" port: 46377 } } peers { permanent_uuid: "a610668bf3dc4846a8f809bc0f652443" member_type: VOTER last_known_addr { host: "127.21.12.188" port: 39245 } }
I20260806 06:28:53.745261 9032 sys_catalog.cc:564] T 00000000000000000000000000000000 P 808c97503f2e438fbbf351cb34758102 [sys.catalog]: configured and running, proceeding with master startup.
W20260806 06:28:53.745755 8959 tablet.cc:2369] T 00000000000000000000000000000000 P 808c97503f2e438fbbf351cb34758102: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 06:28:53.745220 9205 sys_catalog.cc:455] T 00000000000000000000000000000000 P 808c97503f2e438fbbf351cb34758102 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81239de6d9b14d648ebebd041840a733" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 40733 } } peers { permanent_uuid: "808c97503f2e438fbbf351cb34758102" member_type: VOTER last_known_addr { host: "127.21.12.189" port: 46377 } } peers { permanent_uuid: "a610668bf3dc4846a8f809bc0f652443" member_type: VOTER last_known_addr { host: "127.21.12.188" port: 39245 } } }
W20260806 06:28:53.747450 9216 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 808c97503f2e438fbbf351cb34758102: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 06:28:53.747560 9216 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 06:28:53.749275 8922 master_runner.cc:263] No leader master found from master 808c97503f2e438fbbf351cb34758102
I20260806 06:28:53.749327 8922 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260806 06:28:53.749578 9205 sys_catalog.cc:458] T 00000000000000000000000000000000 P 808c97503f2e438fbbf351cb34758102 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 06:28:53.834961 9205 raft_consensus.cc:482] T 00000000000000000000000000000000 P 808c97503f2e438fbbf351cb34758102 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 06:28:53.835103 9205 raft_consensus.cc:504] T 00000000000000000000000000000000 P 808c97503f2e438fbbf351cb34758102 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81239de6d9b14d648ebebd041840a733" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 40733 } } peers { permanent_uuid: "808c97503f2e438fbbf351cb34758102" member_type: VOTER last_known_addr { host: "127.21.12.189" port: 46377 } } peers { permanent_uuid: "a610668bf3dc4846a8f809bc0f652443" member_type: VOTER last_known_addr { host: "127.21.12.188" port: 39245 } }
I20260806 06:28:53.835361 9205 leader_election.cc:288] T 00000000000000000000000000000000 P 808c97503f2e438fbbf351cb34758102 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 81239de6d9b14d648ebebd041840a733 (127.21.12.190:40733), a610668bf3dc4846a8f809bc0f652443 (127.21.12.188:39245)
I20260806 06:28:53.835613 8880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "808c97503f2e438fbbf351cb34758102" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "81239de6d9b14d648ebebd041840a733" is_pre_election: true
I20260806 06:28:53.835631 9107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "808c97503f2e438fbbf351cb34758102" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a610668bf3dc4846a8f809bc0f652443" is_pre_election: true
I20260806 06:28:53.835778 8880 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 81239de6d9b14d648ebebd041840a733 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808c97503f2e438fbbf351cb34758102 in term 0.
I20260806 06:28:53.835781 9107 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a610668bf3dc4846a8f809bc0f652443 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808c97503f2e438fbbf351cb34758102 in term 0.
I20260806 06:28:53.836000 8951 leader_election.cc:302] T 00000000000000000000000000000000 P 808c97503f2e438fbbf351cb34758102 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808c97503f2e438fbbf351cb34758102, 81239de6d9b14d648ebebd041840a733; no voters:
I20260806 06:28:53.836146 9205 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 808c97503f2e438fbbf351cb34758102 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 06:28:53.836225 9205 raft_consensus.cc:482] T 00000000000000000000000000000000 P 808c97503f2e438fbbf351cb34758102 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 06:28:53.836251 9205 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 808c97503f2e438fbbf351cb34758102 [term 0 FOLLOWER]: Advancing to term 1
I20260806 06:28:53.837194 9205 raft_consensus.cc:504] T 00000000000000000000000000000000 P 808c97503f2e438fbbf351cb34758102 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81239de6d9b14d648ebebd041840a733" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 40733 } } peers { permanent_uuid: "808c97503f2e438fbbf351cb34758102" member_type: VOTER last_known_addr { host: "127.21.12.189" port: 46377 } } peers { permanent_uuid: "a610668bf3dc4846a8f809bc0f652443" member_type: VOTER last_known_addr { host: "127.21.12.188" port: 39245 } }
I20260806 06:28:53.837337 9205 leader_election.cc:288] T 00000000000000000000000000000000 P 808c97503f2e438fbbf351cb34758102 [CANDIDATE]: Term 1 election: Requested vote from peers 81239de6d9b14d648ebebd041840a733 (127.21.12.190:40733), a610668bf3dc4846a8f809bc0f652443 (127.21.12.188:39245)
I20260806 06:28:53.837468 9107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "808c97503f2e438fbbf351cb34758102" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a610668bf3dc4846a8f809bc0f652443"
I20260806 06:28:53.837479 8880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "808c97503f2e438fbbf351cb34758102" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "81239de6d9b14d648ebebd041840a733"
I20260806 06:28:53.837556 8880 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 81239de6d9b14d648ebebd041840a733 [term 0 FOLLOWER]: Advancing to term 1
I20260806 06:28:53.837558 9107 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a610668bf3dc4846a8f809bc0f652443 [term 0 FOLLOWER]: Advancing to term 1
I20260806 06:28:53.838187 8880 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 81239de6d9b14d648ebebd041840a733 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808c97503f2e438fbbf351cb34758102 in term 1.
I20260806 06:28:53.838363 8951 leader_election.cc:302] T 00000000000000000000000000000000 P 808c97503f2e438fbbf351cb34758102 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808c97503f2e438fbbf351cb34758102, 81239de6d9b14d648ebebd041840a733; no voters:
I20260806 06:28:53.838435 9107 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a610668bf3dc4846a8f809bc0f652443 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808c97503f2e438fbbf351cb34758102 in term 1.
I20260806 06:28:53.838539 9205 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 808c97503f2e438fbbf351cb34758102 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 06:28:53.838732 9205 raft_consensus.cc:686] T 00000000000000000000000000000000 P 808c97503f2e438fbbf351cb34758102 [term 1 LEADER]: Becoming Leader. State: Replica: 808c97503f2e438fbbf351cb34758102, State: Running, Role: LEADER
I20260806 06:28:53.838832 9205 consensus_queue.cc:237] T 00000000000000000000000000000000 P 808c97503f2e438fbbf351cb34758102 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81239de6d9b14d648ebebd041840a733" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 40733 } } peers { permanent_uuid: "808c97503f2e438fbbf351cb34758102" member_type: VOTER last_known_addr { host: "127.21.12.189" port: 46377 } } peers { permanent_uuid: "a610668bf3dc4846a8f809bc0f652443" member_type: VOTER last_known_addr { host: "127.21.12.188" port: 39245 } }
I20260806 06:28:53.839303 9219 sys_catalog.cc:455] T 00000000000000000000000000000000 P 808c97503f2e438fbbf351cb34758102 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 808c97503f2e438fbbf351cb34758102. Latest consensus state: current_term: 1 leader_uuid: "808c97503f2e438fbbf351cb34758102" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81239de6d9b14d648ebebd041840a733" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 40733 } } peers { permanent_uuid: "808c97503f2e438fbbf351cb34758102" member_type: VOTER last_known_addr { host: "127.21.12.189" port: 46377 } } peers { permanent_uuid: "a610668bf3dc4846a8f809bc0f652443" member_type: VOTER last_known_addr { host: "127.21.12.188" port: 39245 } } }
I20260806 06:28:53.839393 9219 sys_catalog.cc:458] T 00000000000000000000000000000000 P 808c97503f2e438fbbf351cb34758102 [sys.catalog]: This master's current role is: LEADER
I20260806 06:28:53.839609 9222 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 06:28:53.839807 9222 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 06:28:53.841118 8880 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 81239de6d9b14d648ebebd041840a733 [term 1 FOLLOWER]: Refusing update from remote peer 808c97503f2e438fbbf351cb34758102: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 06:28:53.841238 9107 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a610668bf3dc4846a8f809bc0f652443 [term 1 FOLLOWER]: Refusing update from remote peer 808c97503f2e438fbbf351cb34758102: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 06:28:53.841408 9219 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 808c97503f2e438fbbf351cb34758102 [LEADER]: Connected to new peer: Peer: permanent_uuid: "81239de6d9b14d648ebebd041840a733" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 40733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 06:28:53.841553 9205 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 808c97503f2e438fbbf351cb34758102 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a610668bf3dc4846a8f809bc0f652443" member_type: VOTER last_known_addr { host: "127.21.12.188" port: 39245 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260806 06:28:53.842437 8847 tablet.cc:2369] T 00000000000000000000000000000000 P 81239de6d9b14d648ebebd041840a733: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 06:28:53.842672 9180 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81239de6d9b14d648ebebd041840a733 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 808c97503f2e438fbbf351cb34758102. Latest consensus state: current_term: 1 leader_uuid: "808c97503f2e438fbbf351cb34758102" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81239de6d9b14d648ebebd041840a733" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 40733 } } peers { permanent_uuid: "808c97503f2e438fbbf351cb34758102" member_type: VOTER last_known_addr { host: "127.21.12.189" port: 46377 } } peers { permanent_uuid: "a610668bf3dc4846a8f809bc0f652443" member_type: VOTER last_known_addr { host: "127.21.12.188" port: 39245 } } }
I20260806 06:28:53.842754 9180 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81239de6d9b14d648ebebd041840a733 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 06:28:53.842834 9161 sys_catalog.cc:455] T 00000000000000000000000000000000 P a610668bf3dc4846a8f809bc0f652443 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 808c97503f2e438fbbf351cb34758102. Latest consensus state: current_term: 1 leader_uuid: "808c97503f2e438fbbf351cb34758102" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81239de6d9b14d648ebebd041840a733" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 40733 } } peers { permanent_uuid: "808c97503f2e438fbbf351cb34758102" member_type: VOTER last_known_addr { host: "127.21.12.189" port: 46377 } } peers { permanent_uuid: "a610668bf3dc4846a8f809bc0f652443" member_type: VOTER last_known_addr { host: "127.21.12.188" port: 39245 } } }
I20260806 06:28:53.842916 9161 sys_catalog.cc:458] T 00000000000000000000000000000000 P a610668bf3dc4846a8f809bc0f652443 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 06:28:53.843837 9161 sys_catalog.cc:455] T 00000000000000000000000000000000 P a610668bf3dc4846a8f809bc0f652443 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "808c97503f2e438fbbf351cb34758102" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81239de6d9b14d648ebebd041840a733" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 40733 } } peers { permanent_uuid: "808c97503f2e438fbbf351cb34758102" member_type: VOTER last_known_addr { host: "127.21.12.189" port: 46377 } } peers { permanent_uuid: "a610668bf3dc4846a8f809bc0f652443" member_type: VOTER last_known_addr { host: "127.21.12.188" port: 39245 } } }
I20260806 06:28:53.843919 9161 sys_catalog.cc:458] T 00000000000000000000000000000000 P a610668bf3dc4846a8f809bc0f652443 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 06:28:53.843904 9219 sys_catalog.cc:455] T 00000000000000000000000000000000 P 808c97503f2e438fbbf351cb34758102 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "808c97503f2e438fbbf351cb34758102" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81239de6d9b14d648ebebd041840a733" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 40733 } } peers { permanent_uuid: "808c97503f2e438fbbf351cb34758102" member_type: VOTER last_known_addr { host: "127.21.12.189" port: 46377 } } peers { permanent_uuid: "a610668bf3dc4846a8f809bc0f652443" member_type: VOTER last_known_addr { host: "127.21.12.188" port: 39245 } } }
I20260806 06:28:53.843982 9219 sys_catalog.cc:458] T 00000000000000000000000000000000 P 808c97503f2e438fbbf351cb34758102 [sys.catalog]: This master's current role is: LEADER
I20260806 06:28:53.844347 9222 catalog_manager.cc:1347] Generated new cluster ID: 6051b9c5cc8e4d50b6973ed0a9ff4f39
I20260806 06:28:53.844384 9222 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 06:28:53.846691 9228 mvcc.cc:205] Tried to move back new op lower bound from 7315446717811417088 to 7315446717804498944. Current Snapshot: MvccSnapshot[applied={T|T < 7315446717811417088}]
I20260806 06:28:53.850633 9205 sys_catalog.cc:455] T 00000000000000000000000000000000 P 808c97503f2e438fbbf351cb34758102 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "808c97503f2e438fbbf351cb34758102" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81239de6d9b14d648ebebd041840a733" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 40733 } } peers { permanent_uuid: "808c97503f2e438fbbf351cb34758102" member_type: VOTER last_known_addr { host: "127.21.12.189" port: 46377 } } peers { permanent_uuid: "a610668bf3dc4846a8f809bc0f652443" member_type: VOTER last_known_addr { host: "127.21.12.188" port: 39245 } } }
I20260806 06:28:53.850723 9205 sys_catalog.cc:458] T 00000000000000000000000000000000 P 808c97503f2e438fbbf351cb34758102 [sys.catalog]: This master's current role is: LEADER
I20260806 06:28:53.851227 9180 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81239de6d9b14d648ebebd041840a733 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "808c97503f2e438fbbf351cb34758102" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81239de6d9b14d648ebebd041840a733" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 40733 } } peers { permanent_uuid: "808c97503f2e438fbbf351cb34758102" member_type: VOTER last_known_addr { host: "127.21.12.189" port: 46377 } } peers { permanent_uuid: "a610668bf3dc4846a8f809bc0f652443" member_type: VOTER last_known_addr { host: "127.21.12.188" port: 39245 } } }
I20260806 06:28:53.851311 9180 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81239de6d9b14d648ebebd041840a733 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 06:28:53.852066 9222 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 06:28:53.852756 9222 catalog_manager.cc:1504] Loading token signing keys...
I20260806 06:28:53.862613 9222 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 808c97503f2e438fbbf351cb34758102: Generated new TSK 0
I20260806 06:28:53.862814 9222 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 06:28:54.678702 9176 catalog_manager.cc:1259] Loaded cluster ID: 6051b9c5cc8e4d50b6973ed0a9ff4f39
I20260806 06:28:54.678779 9176 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a610668bf3dc4846a8f809bc0f652443: loading cluster ID for follower catalog manager: success
I20260806 06:28:54.679541 9176 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a610668bf3dc4846a8f809bc0f652443: acquiring CA information for follower catalog manager: success
I20260806 06:28:54.679800 9176 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a610668bf3dc4846a8f809bc0f652443: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260806 06:28:54.697129 9202 catalog_manager.cc:1259] Loaded cluster ID: 6051b9c5cc8e4d50b6973ed0a9ff4f39
I20260806 06:28:54.697202 9202 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 81239de6d9b14d648ebebd041840a733: loading cluster ID for follower catalog manager: success
I20260806 06:28:54.698001 9202 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 81239de6d9b14d648ebebd041840a733: acquiring CA information for follower catalog manager: success
I20260806 06:28:54.698359 9202 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 81239de6d9b14d648ebebd041840a733: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260806 06:28:54.927179 21554 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHhLhT/build/release/bin/kudu with pid 8816
I20260806 06:28:54.936370 21554 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHhLhT/build/release/bin/kudu with pid 8922
I20260806 06:28:54.944375 21554 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHhLhT/build/release/bin/kudu with pid 9044
2026-08-06T06:28:54Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (2953 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785997581049705-21554-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 06:28:55 dist-test-slave-4tdj krb5kdc[9244](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 06:28:55 dist-test-slave-4tdj krb5kdc[9244](info): set up 2 sockets
Aug 06 06:28:55 dist-test-slave-4tdj krb5kdc[9244](info): commencing operation
krb5kdc: starting...
W20260806 06:28:56.068742 21554 mini_kdc.cc:121] Time spent starting KDC: real 1.098s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785997581049705-21554-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785997581049705-21554-0/krb5kdc/test-user.keytab.
Aug 06 06:28:56 dist-test-slave-4tdj krb5kdc[9244](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997736, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T06:28:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T06:28:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.12.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.12.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.12.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.190.keytab.
Entry for principal kudu/127.21.12.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.190.keytab.
I20260806 06:28:56.178552 21554 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHhLhT/build/release/bin/kudu
/tmp/dist-test-taskFHhLhT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785997581049705-21554-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785997581049705-21554-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785997581049705-21554-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785997581049705-21554-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.12.190:42551
--webserver_interface=127.21.12.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.190.keytab
--principal=kudu/127.21.12.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.148:33569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.12.190:42551
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785997581049705-21554-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785997581049705-21554-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785997581049705-21554-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 06:28:56.256330 9264 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 06:28:56.256510 9264 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 06:28:56.256531 9264 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 06:28:56.257570 9264 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 06:28:56.257601 9264 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 06:28:56.257614 9264 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 06:28:56.257625 9264 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 06:28:56.257635 9264 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 06:28:56.259040 9264 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.148:33569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785997581049705-21554-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785997581049705-21554-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.12.190:42551
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.190.keytab
--principal=kudu/127.21.12.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.12.190:42551
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785997581049705-21554-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.12.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9264
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785997581049705-21554-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 05:11:02 UTC on ccycloud.quasar-hvzbbz.root.comops.site
build id 4569617
I20260806 06:28:56.259227 9264 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 06:28:56.259397 9264 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 06:28:56.263893 9270 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:28:56.263970 9271 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:28:56.263895 9278 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:28:56.264043 9264 server_base.cc:1034] running on GCE node
I20260806 06:28:56.264369 9264 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 06:28:56.264657 9264 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 06:28:56.265805 9264 hybrid_clock.cc:648] HybridClock initialized: now 1785997736265782 us; error 39 us; skew 500 ppm
Aug 06 06:28:56 dist-test-slave-4tdj krb5kdc[9244](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997736, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 06:28:56.268249 9264 init.cc:377] Logged in from keytab as kudu/127.21.12.190@KRBTEST.COM (short username kudu)
I20260806 06:28:56.268647 9264 webserver.cc:460] Webserver started at http://127.21.12.190:38627/ using document root <none> and password file <none>
I20260806 06:28:56.268859 9264 fs_manager.cc:359] Metadata directory not provided
I20260806 06:28:56.268905 9264 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 06:28:56.269034 9264 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 06:28:56.270056 9264 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785997581049705-21554-0/minicluster-data/master-0/data/instance:
uuid: "0713fdef423e4655a7c7479c03ecf061"
format_stamp: "Formatted at 2026-08-06 06:28:56 on dist-test-slave-4tdj"
server_key: "16d027328f5d71ebb52d21f2b656855f"
server_key_iv: "10477413cc6bbbb1d9ea1eb528b490de"
server_key_version: "encryptionkey@0"
I20260806 06:28:56.270401 9264 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785997581049705-21554-0/minicluster-data/master-0/wal/instance:
uuid: "0713fdef423e4655a7c7479c03ecf061"
format_stamp: "Formatted at 2026-08-06 06:28:56 on dist-test-slave-4tdj"
server_key: "16d027328f5d71ebb52d21f2b656855f"
server_key_iv: "10477413cc6bbbb1d9ea1eb528b490de"
server_key_version: "encryptionkey@0"
I20260806 06:28:56.271602 9264 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260806 06:28:56.272228 9286 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 06:28:56.272374 9264 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260806 06:28:56.272461 9264 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785997581049705-21554-0/minicluster-data/master-0/data,/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785997581049705-21554-0/minicluster-data/master-0/wal
uuid: "0713fdef423e4655a7c7479c03ecf061"
format_stamp: "Formatted at 2026-08-06 06:28:56 on dist-test-slave-4tdj"
server_key: "16d027328f5d71ebb52d21f2b656855f"
server_key_iv: "10477413cc6bbbb1d9ea1eb528b490de"
server_key_version: "encryptionkey@0"
I20260806 06:28:56.272518 9264 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785997581049705-21554-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785997581049705-21554-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785997581049705-21554-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 06:28:56.284544 9264 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 06:28:56.286108 9264 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 06:28:56.286232 9264 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 06:28:56.289758 9264 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.190:42551
I20260806 06:28:56.289788 9366 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.190:42551 every 8 connection(s)
I20260806 06:28:56.290083 9264 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785997581049705-21554-0/minicluster-data/master-0/data/info.pb
I20260806 06:28:56.290938 9368 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 06:28:56.292577 21554 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHhLhT/build/release/bin/kudu as pid 9264
I20260806 06:28:56.292675 21554 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785997581049705-21554-0/minicluster-data/master-0/wal/instance
I20260806 06:28:56.292877 21554 external_mini_cluster.cc:1428] Setting key 3cfa0d18a5775bc19f070bd89c7caf75
I20260806 06:28:56.293233 9368 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0713fdef423e4655a7c7479c03ecf061: Bootstrap starting.
I20260806 06:28:56.293869 9368 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0713fdef423e4655a7c7479c03ecf061: Neither blocks nor log segments found. Creating new log.
I20260806 06:28:56.294180 9368 log.cc:824] T 00000000000000000000000000000000 P 0713fdef423e4655a7c7479c03ecf061: Log is configured to *not* fsync() on all Append() calls
I20260806 06:28:56.294873 9368 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0713fdef423e4655a7c7479c03ecf061: No bootstrap required, opened a new log
I20260806 06:28:56.297020 9368 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0713fdef423e4655a7c7479c03ecf061 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0713fdef423e4655a7c7479c03ecf061" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 42551 } }
I20260806 06:28:56.297163 9368 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0713fdef423e4655a7c7479c03ecf061 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 06:28:56.297209 9368 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0713fdef423e4655a7c7479c03ecf061 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0713fdef423e4655a7c7479c03ecf061, State: Initialized, Role: FOLLOWER
I20260806 06:28:56.297322 9368 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0713fdef423e4655a7c7479c03ecf061 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0713fdef423e4655a7c7479c03ecf061" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 42551 } }
I20260806 06:28:56.297389 9368 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0713fdef423e4655a7c7479c03ecf061 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 06:28:56.297423 9368 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0713fdef423e4655a7c7479c03ecf061 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 06:28:56.297465 9368 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0713fdef423e4655a7c7479c03ecf061 [term 0 FOLLOWER]: Advancing to term 1
I20260806 06:28:56.298079 9368 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0713fdef423e4655a7c7479c03ecf061 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0713fdef423e4655a7c7479c03ecf061" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 42551 } }
I20260806 06:28:56.298184 9368 leader_election.cc:302] T 00000000000000000000000000000000 P 0713fdef423e4655a7c7479c03ecf061 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0713fdef423e4655a7c7479c03ecf061; no voters:
I20260806 06:28:56.298360 9368 leader_election.cc:288] T 00000000000000000000000000000000 P 0713fdef423e4655a7c7479c03ecf061 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 06:28:56.298398 9378 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0713fdef423e4655a7c7479c03ecf061 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 06:28:56.298537 9378 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0713fdef423e4655a7c7479c03ecf061 [term 1 LEADER]: Becoming Leader. State: Replica: 0713fdef423e4655a7c7479c03ecf061, State: Running, Role: LEADER
I20260806 06:28:56.298614 9368 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0713fdef423e4655a7c7479c03ecf061 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 06:28:56.298631 9378 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0713fdef423e4655a7c7479c03ecf061 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0713fdef423e4655a7c7479c03ecf061" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 42551 } }
I20260806 06:28:56.299016 9382 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0713fdef423e4655a7c7479c03ecf061 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0713fdef423e4655a7c7479c03ecf061. Latest consensus state: current_term: 1 leader_uuid: "0713fdef423e4655a7c7479c03ecf061" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0713fdef423e4655a7c7479c03ecf061" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 42551 } } }
I20260806 06:28:56.299109 9382 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0713fdef423e4655a7c7479c03ecf061 [sys.catalog]: This master's current role is: LEADER
I20260806 06:28:56.299029 9379 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0713fdef423e4655a7c7479c03ecf061 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0713fdef423e4655a7c7479c03ecf061" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0713fdef423e4655a7c7479c03ecf061" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 42551 } } }
I20260806 06:28:56.299153 9379 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0713fdef423e4655a7c7479c03ecf061 [sys.catalog]: This master's current role is: LEADER
I20260806 06:28:56.299438 9391 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 06:28:56.299927 9391 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Aug 06 06:28:56 dist-test-slave-4tdj krb5kdc[9244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997736, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.12.190@KRBTEST.COM
I20260806 06:28:56.301465 9391 catalog_manager.cc:1347] Generated new cluster ID: 878ebe6be98e41afb0bbdf968f0c94f6
I20260806 06:28:56.301515 9391 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 06:28:56.303488 9376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:28:56.293367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53964 (local address 127.21.12.190:42551)
0806 06:28:56.293545 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0806 06:28:56.293555 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 06:28:56.293591 (+ 36us) server_negotiation.cc:407] Connection header received
0806 06:28:56.294129 (+ 538us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:28:56.294138 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:28:56.294211 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:28:56.294401 (+ 190us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:28:56.295063 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:28:56.296416 (+ 1353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:28:56.297119 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:28:56.297711 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:28:56.297818 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:28:56.300473 (+ 2655us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:28:56.300476 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:28:56.300476 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 06:28:56.300485 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:28:56.302085 (+ 1600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:28:56.302445 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:28:56.302449 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:28:56.302449 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:28:56.302530 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:28:56.302704 (+ 174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:28:56.302705 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:28:56.302706 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:28:56.302920 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:28:56.303059 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0806 06:28:56.303174 (+ 115us) server_negotiation.cc:299] Negotiation successful
0806 06:28:56.303303 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":124,"thread_start_us":79,"threads_started":1}
I20260806 06:28:56.307243 21554 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 06:28:56.317498 9391 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 06:28:56.318008 9391 catalog_manager.cc:1504] Loading token signing keys...
I20260806 06:28:56.328033 9391 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0713fdef423e4655a7c7479c03ecf061: Generated new TSK 0
I20260806 06:28:56.328195 9391 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 06:28:56.360841 21554 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHhLhT/build/release/bin/kudu with pid 9264
2026-08-06T06:28:56Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1414 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 06:28:56 dist-test-slave-4tdj krb5kdc[9417](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 06:28:56 dist-test-slave-4tdj krb5kdc[9417](info): set up 2 sockets
Aug 06 06:28:56 dist-test-slave-4tdj krb5kdc[9417](info): commencing operation
krb5kdc: starting...
W20260806 06:28:57.493206 21554 mini_kdc.cc:121] Time spent starting KDC: real 1.109s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/krb5kdc/test-user.keytab.
Aug 06 06:28:57 dist-test-slave-4tdj krb5kdc[9417](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997737, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T06:28:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T06:28:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.12.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.12.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.12.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.190.keytab.
Entry for principal kudu/127.21.12.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.190.keytab.
I20260806 06:28:57.604923 21554 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHhLhT/build/release/bin/kudu
/tmp/dist-test-taskFHhLhT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.12.190:36809
--webserver_interface=127.21.12.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.190.keytab
--principal=kudu/127.21.12.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.148:41671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.12.190:36809
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 06:28:57.684278 9436 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 06:28:57.684477 9436 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 06:28:57.684506 9436 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 06:28:57.685621 9436 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 06:28:57.685658 9436 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 06:28:57.685678 9436 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 06:28:57.685693 9436 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 06:28:57.685708 9436 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 06:28:57.687223 9436 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.148:41671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.12.190:36809
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.190.keytab
--principal=kudu/127.21.12.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.12.190:36809
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.12.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9436
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 05:11:02 UTC on ccycloud.quasar-hvzbbz.root.comops.site
build id 4569617
I20260806 06:28:57.687465 9436 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 06:28:57.687661 9436 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 06:28:57.692438 9442 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:28:57.692469 9446 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:28:57.692409 9444 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:28:57.692691 9436 server_base.cc:1034] running on GCE node
I20260806 06:28:57.692903 9436 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 06:28:57.693161 9436 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 06:28:57.694326 9436 hybrid_clock.cc:648] HybridClock initialized: now 1785997737694313 us; error 29 us; skew 500 ppm
Aug 06 06:28:57 dist-test-slave-4tdj krb5kdc[9417](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997737, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 06:28:57.703015 9436 init.cc:377] Logged in from keytab as kudu/127.21.12.190@KRBTEST.COM (short username kudu)
I20260806 06:28:57.703493 9436 webserver.cc:460] Webserver started at http://127.21.12.190:44209/ using document root <none> and password file <none>
I20260806 06:28:57.703715 9436 fs_manager.cc:359] Metadata directory not provided
I20260806 06:28:57.703763 9436 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 06:28:57.703887 9436 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 06:28:57.704879 9436 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/minicluster-data/master-0/data/instance:
uuid: "2ad50aad27464833af0c85627af4d871"
format_stamp: "Formatted at 2026-08-06 06:28:57 on dist-test-slave-4tdj"
server_key: "e7f5ee43b1a59a3d790275abcf9c16b2"
server_key_iv: "4be3aafe4a3837f23cfd543aa9b2c504"
server_key_version: "encryptionkey@0"
I20260806 06:28:57.705225 9436 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/minicluster-data/master-0/wal/instance:
uuid: "2ad50aad27464833af0c85627af4d871"
format_stamp: "Formatted at 2026-08-06 06:28:57 on dist-test-slave-4tdj"
server_key: "e7f5ee43b1a59a3d790275abcf9c16b2"
server_key_iv: "4be3aafe4a3837f23cfd543aa9b2c504"
server_key_version: "encryptionkey@0"
I20260806 06:28:57.706482 9436 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260806 06:28:57.707144 9455 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 06:28:57.707291 9436 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260806 06:28:57.707376 9436 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/minicluster-data/master-0/data,/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/minicluster-data/master-0/wal
uuid: "2ad50aad27464833af0c85627af4d871"
format_stamp: "Formatted at 2026-08-06 06:28:57 on dist-test-slave-4tdj"
server_key: "e7f5ee43b1a59a3d790275abcf9c16b2"
server_key_iv: "4be3aafe4a3837f23cfd543aa9b2c504"
server_key_version: "encryptionkey@0"
I20260806 06:28:57.707430 9436 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 06:28:57.729794 9436 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 06:28:57.731781 9436 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 06:28:57.731910 9436 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 06:28:57.735524 9436 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.190:36809
I20260806 06:28:57.735562 9534 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.190:36809 every 8 connection(s)
I20260806 06:28:57.735867 9436 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/minicluster-data/master-0/data/info.pb
I20260806 06:28:57.736771 9535 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 06:28:57.738933 9535 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ad50aad27464833af0c85627af4d871: Bootstrap starting.
I20260806 06:28:57.739246 21554 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHhLhT/build/release/bin/kudu as pid 9436
I20260806 06:28:57.739341 21554 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/minicluster-data/master-0/wal/instance
I20260806 06:28:57.739475 9535 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2ad50aad27464833af0c85627af4d871: Neither blocks nor log segments found. Creating new log.
I20260806 06:28:57.739504 21554 external_mini_cluster.cc:1428] Setting key cddfc4699b8fb01753285f81e5b63c98
I20260806 06:28:57.739738 9535 log.cc:824] T 00000000000000000000000000000000 P 2ad50aad27464833af0c85627af4d871: Log is configured to *not* fsync() on all Append() calls
I20260806 06:28:57.740487 9535 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ad50aad27464833af0c85627af4d871: No bootstrap required, opened a new log
I20260806 06:28:57.742703 9535 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2ad50aad27464833af0c85627af4d871 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ad50aad27464833af0c85627af4d871" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 36809 } }
I20260806 06:28:57.742863 9535 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2ad50aad27464833af0c85627af4d871 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 06:28:57.742913 9535 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2ad50aad27464833af0c85627af4d871 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ad50aad27464833af0c85627af4d871, State: Initialized, Role: FOLLOWER
I20260806 06:28:57.743036 9535 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2ad50aad27464833af0c85627af4d871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ad50aad27464833af0c85627af4d871" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 36809 } }
I20260806 06:28:57.743119 9535 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2ad50aad27464833af0c85627af4d871 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 06:28:57.743150 9535 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2ad50aad27464833af0c85627af4d871 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 06:28:57.743204 9535 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2ad50aad27464833af0c85627af4d871 [term 0 FOLLOWER]: Advancing to term 1
I20260806 06:28:57.743719 9535 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2ad50aad27464833af0c85627af4d871 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ad50aad27464833af0c85627af4d871" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 36809 } }
I20260806 06:28:57.743820 9535 leader_election.cc:302] T 00000000000000000000000000000000 P 2ad50aad27464833af0c85627af4d871 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2ad50aad27464833af0c85627af4d871; no voters:
I20260806 06:28:57.743961 9535 leader_election.cc:288] T 00000000000000000000000000000000 P 2ad50aad27464833af0c85627af4d871 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 06:28:57.744000 9542 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2ad50aad27464833af0c85627af4d871 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 06:28:57.744145 9542 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2ad50aad27464833af0c85627af4d871 [term 1 LEADER]: Becoming Leader. State: Replica: 2ad50aad27464833af0c85627af4d871, State: Running, Role: LEADER
I20260806 06:28:57.744200 9535 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2ad50aad27464833af0c85627af4d871 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 06:28:57.744231 9542 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2ad50aad27464833af0c85627af4d871 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ad50aad27464833af0c85627af4d871" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 36809 } }
I20260806 06:28:57.744616 9544 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ad50aad27464833af0c85627af4d871 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2ad50aad27464833af0c85627af4d871. Latest consensus state: current_term: 1 leader_uuid: "2ad50aad27464833af0c85627af4d871" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ad50aad27464833af0c85627af4d871" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 36809 } } }
I20260806 06:28:57.744690 9544 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ad50aad27464833af0c85627af4d871 [sys.catalog]: This master's current role is: LEADER
I20260806 06:28:57.744855 9543 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ad50aad27464833af0c85627af4d871 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2ad50aad27464833af0c85627af4d871" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ad50aad27464833af0c85627af4d871" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 36809 } } }
I20260806 06:28:57.744907 9543 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ad50aad27464833af0c85627af4d871 [sys.catalog]: This master's current role is: LEADER
I20260806 06:28:57.745074 9554 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 06:28:57.745590 9554 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Aug 06 06:28:57 dist-test-slave-4tdj krb5kdc[9417](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997737, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.12.190@KRBTEST.COM
I20260806 06:28:57.747144 9554 catalog_manager.cc:1347] Generated new cluster ID: 377b98b622474c4ab577dd247b3d558d
I20260806 06:28:57.747202 9554 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 06:28:57.750568 9541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:28:57.739980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45934 (local address 127.21.12.190:36809)
0806 06:28:57.740138 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0806 06:28:57.740145 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 06:28:57.740175 (+ 30us) server_negotiation.cc:407] Connection header received
0806 06:28:57.740728 (+ 553us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:28:57.740737 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:28:57.740796 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:28:57.740938 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:28:57.741472 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:28:57.742802 (+ 1330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:28:57.743629 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:28:57.744224 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:28:57.744624 (+ 400us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:28:57.746999 (+ 2375us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:28:57.747003 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:28:57.747003 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 06:28:57.747015 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:28:57.748801 (+ 1786us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:28:57.749150 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:28:57.749152 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:28:57.749153 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:28:57.749244 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:28:57.749408 (+ 164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:28:57.749408 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:28:57.749408 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:28:57.749661 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:28:57.750161 (+ 500us) server_negotiation.cc:1035] Waiting for connection context
0806 06:28:57.750226 (+ 65us) server_negotiation.cc:299] Negotiation successful
0806 06:28:57.750384 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":72,"threads_started":1}
I20260806 06:28:57.757527 9554 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 06:28:57.758220 9554 catalog_manager.cc:1504] Loading token signing keys...
I20260806 06:28:57.765842 9554 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2ad50aad27464833af0c85627af4d871: Generated new TSK 0
I20260806 06:28:57.766002 9554 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.12.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.12.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.12.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.129.keytab.
Entry for principal kudu/127.21.12.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.129.keytab.
I20260806 06:28:57.790326 21554 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHhLhT/build/release/bin/kudu
/tmp/dist-test-taskFHhLhT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.12.129:0
--local_ip_for_outbound_sockets=127.21.12.129
--webserver_interface=127.21.12.129
--webserver_port=0
--tserver_master_addrs=127.21.12.190:36809
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.129.keytab
--principal=kudu/127.21.12.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.148:41671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 06:28:57.869652 9575 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 06:28:57.869860 9575 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 06:28:57.869881 9575 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 06:28:57.871053 9575 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 06:28:57.871096 9575 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 06:28:57.871121 9575 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.12.129
I20260806 06:28:57.872543 9575 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.148:41671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.129.keytab
--principal=kudu/127.21.12.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.12.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.12.129
--webserver_port=0
--tserver_master_addrs=127.21.12.190:36809
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9575
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.12.129
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 05:11:02 UTC on ccycloud.quasar-hvzbbz.root.comops.site
build id 4569617
I20260806 06:28:57.872761 9575 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 06:28:57.872973 9575 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 06:28:57.877959 9585 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:28:57.877928 9586 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:28:57.878165 9575 server_base.cc:1034] running on GCE node
W20260806 06:28:57.878417 9590 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:28:57.878623 9575 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 06:28:57.878866 9575 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 06:28:57.880007 9575 hybrid_clock.cc:648] HybridClock initialized: now 1785997737879991 us; error 27 us; skew 500 ppm
Aug 06 06:28:57 dist-test-slave-4tdj krb5kdc[9417](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997737, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 06:28:57.883117 9575 init.cc:377] Logged in from keytab as kudu/127.21.12.129@KRBTEST.COM (short username kudu)
I20260806 06:28:57.883576 9575 webserver.cc:460] Webserver started at http://127.21.12.129:38883/ using document root <none> and password file <none>
I20260806 06:28:57.883778 9575 fs_manager.cc:359] Metadata directory not provided
I20260806 06:28:57.883826 9575 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 06:28:57.883948 9575 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 06:28:57.884979 9575 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/minicluster-data/ts-0/data/instance:
uuid: "e7a2421395c9473e91a329c11588a784"
format_stamp: "Formatted at 2026-08-06 06:28:57 on dist-test-slave-4tdj"
server_key: "e01ae4fd34e538a96a7c18bb93f98cb4"
server_key_iv: "8d4919674b8d1227fbff0693830034c0"
server_key_version: "encryptionkey@0"
I20260806 06:28:57.885334 9575 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/minicluster-data/ts-0/wal/instance:
uuid: "e7a2421395c9473e91a329c11588a784"
format_stamp: "Formatted at 2026-08-06 06:28:57 on dist-test-slave-4tdj"
server_key: "e01ae4fd34e538a96a7c18bb93f98cb4"
server_key_iv: "8d4919674b8d1227fbff0693830034c0"
server_key_version: "encryptionkey@0"
I20260806 06:28:57.886593 9575 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260806 06:28:57.887317 9606 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 06:28:57.887470 9575 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260806 06:28:57.887554 9575 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/minicluster-data/ts-0/wal
uuid: "e7a2421395c9473e91a329c11588a784"
format_stamp: "Formatted at 2026-08-06 06:28:57 on dist-test-slave-4tdj"
server_key: "e01ae4fd34e538a96a7c18bb93f98cb4"
server_key_iv: "8d4919674b8d1227fbff0693830034c0"
server_key_version: "encryptionkey@0"
I20260806 06:28:57.887619 9575 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 06:28:57.911896 9575 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 06:28:57.913988 9575 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 06:28:57.914130 9575 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 06:28:57.914383 9575 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 06:28:57.914695 9575 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 06:28:57.914729 9575 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 06:28:57.914757 9575 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 06:28:57.914774 9575 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 06:28:57.920126 9575 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.129:42271
I20260806 06:28:57.920190 9762 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.129:42271 every 8 connection(s)
I20260806 06:28:57.920480 9575 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/minicluster-data/ts-0/data/info.pb
I20260806 06:28:57.924741 21554 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHhLhT/build/release/bin/kudu as pid 9575
I20260806 06:28:57.924826 21554 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/minicluster-data/ts-0/wal/instance
I20260806 06:28:57.925005 21554 external_mini_cluster.cc:1428] Setting key ca30ced71ecf128340563291b9d3a69e
Aug 06 06:28:57 dist-test-slave-4tdj krb5kdc[9417](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997737, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.129@KRBTEST.COM for kudu/127.21.12.190@KRBTEST.COM
I20260806 06:28:57.929245 9541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:28:57.921131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.129:57881 (local address 127.21.12.190:36809)
0806 06:28:57.921194 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0806 06:28:57.921196 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:28:57.921408 (+ 212us) server_negotiation.cc:407] Connection header received
0806 06:28:57.921884 (+ 476us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:28:57.921885 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:28:57.921896 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:28:57.921949 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:28:57.922902 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:28:57.923397 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:28:57.924288 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:28:57.924485 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:28:57.924548 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:28:57.926700 (+ 2152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:28:57.926701 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:28:57.926702 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 06:28:57.926710 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:28:57.927984 (+ 1274us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:28:57.928420 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:28:57.928420 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:28:57.928421 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:28:57.928465 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:28:57.928694 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:28:57.928695 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:28:57.928695 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:28:57.928789 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:28:57.928832 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0806 06:28:57.929117 (+ 285us) server_negotiation.cc:299] Negotiation successful
0806 06:28:57.929178 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260806 06:28:57.929492 9766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:28:57.921117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:36809 (local address 127.21.12.129:57881)
0806 06:28:57.921297 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0806 06:28:57.921332 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0806 06:28:57.921796 (+ 464us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:28:57.922003 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:28:57.922006 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:28:57.922611 (+ 605us) client_negotiation.cc:189] Negotiated authn=SASL
0806 06:28:57.922803 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:28:57.922808 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:28:57.923463 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:28:57.923464 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:28:57.924230 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:28:57.924232 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:28:57.924536 (+ 304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:28:57.924537 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:28:57.924639 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:28:57.925122 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 06:28:57.925133 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0806 06:28:57.926617 (+ 1484us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 06:28:57.928060 (+ 1443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:28:57.928067 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:28:57.928070 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:28:57.928353 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:28:57.928515 (+ 162us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:28:57.928515 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:28:57.928516 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:28:57.928638 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:28:57.928838 (+ 200us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 06:28:57.928844 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 06:28:57.928992 (+ 148us) client_negotiation.cc:769] Sending connection context
0806 06:28:57.929114 (+ 122us) client_negotiation.cc:240] Negotiation successful
0806 06:28:57.929286 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":104,"threads_started":1}
I20260806 06:28:57.930011 9763 heartbeater.cc:347] Connected to a master server at 127.21.12.190:36809
I20260806 06:28:57.930115 9763 heartbeater.cc:464] Registering TS with master...
I20260806 06:28:57.930335 9763 heartbeater.cc:511] Master 127.21.12.190:36809 requested a full tablet report, sending...
I20260806 06:28:57.930869 9480 ts_manager.cc:194] Registered new tserver with Master: e7a2421395c9473e91a329c11588a784 (127.21.12.129:42271)
I20260806 06:28:57.931464 9480 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.12.129@KRBTEST.COM'} at 127.21.12.129:57881
WARNING: no policy specified for kudu/127.21.12.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.12.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.12.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.130.keytab.
Entry for principal kudu/127.21.12.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.130.keytab.
I20260806 06:28:57.950497 21554 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHhLhT/build/release/bin/kudu
/tmp/dist-test-taskFHhLhT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.12.130:0
--local_ip_for_outbound_sockets=127.21.12.130
--webserver_interface=127.21.12.130
--webserver_port=0
--tserver_master_addrs=127.21.12.190:36809
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.130.keytab
--principal=kudu/127.21.12.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.148:41671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 06:28:58.030560 9771 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 06:28:58.030756 9771 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 06:28:58.030777 9771 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 06:28:58.031893 9771 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 06:28:58.031931 9771 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 06:28:58.031958 9771 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.12.130
I20260806 06:28:58.033380 9771 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.148:41671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.130.keytab
--principal=kudu/127.21.12.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.12.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.12.130
--webserver_port=0
--tserver_master_addrs=127.21.12.190:36809
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9771
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.12.130
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 05:11:02 UTC on ccycloud.quasar-hvzbbz.root.comops.site
build id 4569617
I20260806 06:28:58.033602 9771 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 06:28:58.033809 9771 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 06:28:58.038573 9782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:28:58.038569 9780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:28:58.038740 9771 server_base.cc:1034] running on GCE node
W20260806 06:28:58.038604 9779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:28:58.039004 9771 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 06:28:58.039233 9771 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 06:28:58.040382 9771 hybrid_clock.cc:648] HybridClock initialized: now 1785997738040371 us; error 28 us; skew 500 ppm
Aug 06 06:28:58 dist-test-slave-4tdj krb5kdc[9417](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997738, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 06:28:58.042740 9771 init.cc:377] Logged in from keytab as kudu/127.21.12.130@KRBTEST.COM (short username kudu)
I20260806 06:28:58.043109 9771 webserver.cc:460] Webserver started at http://127.21.12.130:43723/ using document root <none> and password file <none>
I20260806 06:28:58.043282 9771 fs_manager.cc:359] Metadata directory not provided
I20260806 06:28:58.043321 9771 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 06:28:58.043409 9771 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 06:28:58.044379 9771 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/minicluster-data/ts-1/data/instance:
uuid: "a397a32a5c4c4681951c1de266125f9c"
format_stamp: "Formatted at 2026-08-06 06:28:58 on dist-test-slave-4tdj"
server_key: "de85f746c85d5607dfe6f4a622f7693c"
server_key_iv: "0c0d6c91ab3445aad3773bb0edae0806"
server_key_version: "encryptionkey@0"
I20260806 06:28:58.044718 9771 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/minicluster-data/ts-1/wal/instance:
uuid: "a397a32a5c4c4681951c1de266125f9c"
format_stamp: "Formatted at 2026-08-06 06:28:58 on dist-test-slave-4tdj"
server_key: "de85f746c85d5607dfe6f4a622f7693c"
server_key_iv: "0c0d6c91ab3445aad3773bb0edae0806"
server_key_version: "encryptionkey@0"
I20260806 06:28:58.045966 9771 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260806 06:28:58.046643 9792 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 06:28:58.046766 9771 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260806 06:28:58.046846 9771 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/minicluster-data/ts-1/wal
uuid: "a397a32a5c4c4681951c1de266125f9c"
format_stamp: "Formatted at 2026-08-06 06:28:58 on dist-test-slave-4tdj"
server_key: "de85f746c85d5607dfe6f4a622f7693c"
server_key_iv: "0c0d6c91ab3445aad3773bb0edae0806"
server_key_version: "encryptionkey@0"
I20260806 06:28:58.046906 9771 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 06:28:58.060977 9771 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 06:28:58.062819 9771 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 06:28:58.062937 9771 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 06:28:58.063139 9771 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 06:28:58.063433 9771 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 06:28:58.063467 9771 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 06:28:58.063494 9771 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 06:28:58.063511 9771 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 06:28:58.068789 9771 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.130:43341
I20260806 06:28:58.068852 9955 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.130:43341 every 8 connection(s)
I20260806 06:28:58.069111 9771 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/minicluster-data/ts-1/data/info.pb
Aug 06 06:28:58 dist-test-slave-4tdj krb5kdc[9417](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997738, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.130@KRBTEST.COM for kudu/127.21.12.190@KRBTEST.COM
I20260806 06:28:58.075343 21554 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHhLhT/build/release/bin/kudu as pid 9771
I20260806 06:28:58.075452 21554 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/minicluster-data/ts-1/wal/instance
I20260806 06:28:58.075661 21554 external_mini_cluster.cc:1428] Setting key f4afdd6ce2777c2df5ccde8c08dd4316
I20260806 06:28:58.077739 9541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:28:58.069785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.130:37311 (local address 127.21.12.190:36809)
0806 06:28:58.069838 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0806 06:28:58.069840 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:28:58.070013 (+ 173us) server_negotiation.cc:407] Connection header received
0806 06:28:58.070556 (+ 543us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:28:58.070557 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:28:58.070567 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:28:58.070614 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:28:58.071508 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:28:58.072013 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:28:58.072906 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:28:58.073098 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:28:58.073150 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:28:58.075002 (+ 1852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:28:58.075003 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:28:58.075003 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 06:28:58.075010 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:28:58.076507 (+ 1497us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:28:58.076896 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:28:58.076897 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:28:58.076897 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:28:58.076962 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:28:58.077154 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:28:58.077154 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:28:58.077155 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:28:58.077287 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:28:58.077352 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0806 06:28:58.077561 (+ 209us) server_negotiation.cc:299] Negotiation successful
0806 06:28:58.077647 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260806 06:28:58.077795 9959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:28:58.069758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:36809 (local address 127.21.12.130:37311)
0806 06:28:58.069904 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0806 06:28:58.069938 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0806 06:28:58.070472 (+ 534us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:28:58.070673 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:28:58.070677 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:28:58.071234 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0806 06:28:58.071416 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:28:58.071420 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:28:58.072086 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:28:58.072086 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:28:58.072840 (+ 754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:28:58.072841 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:28:58.073154 (+ 313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:28:58.073155 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:28:58.073249 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:28:58.073644 (+ 395us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 06:28:58.073652 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0806 06:28:58.074925 (+ 1273us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 06:28:58.076609 (+ 1684us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:28:58.076614 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:28:58.076616 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:28:58.076831 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:28:58.077020 (+ 189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:28:58.077020 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:28:58.077020 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:28:58.077106 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:28:58.077346 (+ 240us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 06:28:58.077349 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 06:28:58.077450 (+ 101us) client_negotiation.cc:769] Sending connection context
0806 06:28:58.077521 (+ 71us) client_negotiation.cc:240] Negotiation successful
0806 06:28:58.077630 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":85,"threads_started":1}
I20260806 06:28:58.078390 9957 heartbeater.cc:347] Connected to a master server at 127.21.12.190:36809
I20260806 06:28:58.078476 9957 heartbeater.cc:464] Registering TS with master...
I20260806 06:28:58.078636 9957 heartbeater.cc:511] Master 127.21.12.190:36809 requested a full tablet report, sending...
I20260806 06:28:58.078976 9480 ts_manager.cc:194] Registered new tserver with Master: a397a32a5c4c4681951c1de266125f9c (127.21.12.130:43341)
I20260806 06:28:58.079442 9480 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.12.130@KRBTEST.COM'} at 127.21.12.130:37311
WARNING: no policy specified for kudu/127.21.12.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.12.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.12.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.131.keytab.
Entry for principal kudu/127.21.12.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.131.keytab.
I20260806 06:28:58.098636 21554 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHhLhT/build/release/bin/kudu
/tmp/dist-test-taskFHhLhT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.12.131:0
--local_ip_for_outbound_sockets=127.21.12.131
--webserver_interface=127.21.12.131
--webserver_port=0
--tserver_master_addrs=127.21.12.190:36809
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.131.keytab
--principal=kudu/127.21.12.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.148:41671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 06:28:58.177843 9963 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 06:28:58.178040 9963 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 06:28:58.178072 9963 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 06:28:58.179210 9963 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 06:28:58.179250 9963 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 06:28:58.179284 9963 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.12.131
I20260806 06:28:58.180722 9963 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.148:41671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.131.keytab
--principal=kudu/127.21.12.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.12.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.12.131
--webserver_port=0
--tserver_master_addrs=127.21.12.190:36809
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9963
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.12.131
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 05:11:02 UTC on ccycloud.quasar-hvzbbz.root.comops.site
build id 4569617
I20260806 06:28:58.180941 9963 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 06:28:58.181140 9963 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 06:28:58.185811 9969 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:28:58.185822 9974 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:28:58.185869 9971 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:28:58.185995 9963 server_base.cc:1034] running on GCE node
I20260806 06:28:58.186157 9963 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 06:28:58.186364 9963 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 06:28:58.187516 9963 hybrid_clock.cc:648] HybridClock initialized: now 1785997738187507 us; error 27 us; skew 500 ppm
Aug 06 06:28:58 dist-test-slave-4tdj krb5kdc[9417](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997738, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 06:28:58.189755 9963 init.cc:377] Logged in from keytab as kudu/127.21.12.131@KRBTEST.COM (short username kudu)
I20260806 06:28:58.190136 9963 webserver.cc:460] Webserver started at http://127.21.12.131:42145/ using document root <none> and password file <none>
I20260806 06:28:58.190395 9963 fs_manager.cc:359] Metadata directory not provided
I20260806 06:28:58.190447 9963 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 06:28:58.190575 9963 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 06:28:58.191555 9963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/minicluster-data/ts-2/data/instance:
uuid: "b778fbcc33a740b0876dd34c9fa6829c"
format_stamp: "Formatted at 2026-08-06 06:28:58 on dist-test-slave-4tdj"
server_key: "c10e6fb753d0a8555df8b25ce0e3e3da"
server_key_iv: "a2995c2452fb6ac189c123a98f246f68"
server_key_version: "encryptionkey@0"
I20260806 06:28:58.191866 9963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/minicluster-data/ts-2/wal/instance:
uuid: "b778fbcc33a740b0876dd34c9fa6829c"
format_stamp: "Formatted at 2026-08-06 06:28:58 on dist-test-slave-4tdj"
server_key: "c10e6fb753d0a8555df8b25ce0e3e3da"
server_key_iv: "a2995c2452fb6ac189c123a98f246f68"
server_key_version: "encryptionkey@0"
I20260806 06:28:58.193013 9963 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260806 06:28:58.193647 9985 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 06:28:58.193795 9963 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260806 06:28:58.193877 9963 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/minicluster-data/ts-2/wal
uuid: "b778fbcc33a740b0876dd34c9fa6829c"
format_stamp: "Formatted at 2026-08-06 06:28:58 on dist-test-slave-4tdj"
server_key: "c10e6fb753d0a8555df8b25ce0e3e3da"
server_key_iv: "a2995c2452fb6ac189c123a98f246f68"
server_key_version: "encryptionkey@0"
I20260806 06:28:58.193931 9963 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 06:28:58.212634 9963 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 06:28:58.214224 9963 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 06:28:58.214380 9963 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 06:28:58.214612 9963 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 06:28:58.214906 9963 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 06:28:58.214939 9963 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 06:28:58.214967 9963 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 06:28:58.214990 9963 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 06:28:58.220505 9963 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.131:32863
I20260806 06:28:58.220543 10148 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.131:32863 every 8 connection(s)
I20260806 06:28:58.220824 9963 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/minicluster-data/ts-2/data/info.pb
I20260806 06:28:58.223045 21554 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHhLhT/build/release/bin/kudu as pid 9963
I20260806 06:28:58.223165 21554 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/minicluster-data/ts-2/wal/instance
I20260806 06:28:58.223361 21554 external_mini_cluster.cc:1428] Setting key eb24459d79fa827f77d29876cac9c9f0
Aug 06 06:28:58 dist-test-slave-4tdj krb5kdc[9417](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997738, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.131@KRBTEST.COM for kudu/127.21.12.190@KRBTEST.COM
I20260806 06:28:58.229314 9541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:28:58.221476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.131:33371 (local address 127.21.12.190:36809)
0806 06:28:58.221542 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0806 06:28:58.221544 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:28:58.221692 (+ 148us) server_negotiation.cc:407] Connection header received
0806 06:28:58.222155 (+ 463us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:28:58.222156 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:28:58.222166 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:28:58.222220 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:28:58.223158 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:28:58.223826 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:28:58.224943 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:28:58.225133 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:28:58.225187 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:28:58.227177 (+ 1990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:28:58.227179 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:28:58.227179 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 06:28:58.227186 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:28:58.228306 (+ 1120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:28:58.228643 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:28:58.228644 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:28:58.228644 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:28:58.228698 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:28:58.228870 (+ 172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:28:58.228871 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:28:58.228871 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:28:58.228962 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:28:58.229007 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0806 06:28:58.229193 (+ 186us) server_negotiation.cc:299] Negotiation successful
0806 06:28:58.229251 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260806 06:28:58.229482 10154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:28:58.221430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:36809 (local address 127.21.12.131:33371)
0806 06:28:58.221576 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0806 06:28:58.221616 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0806 06:28:58.222077 (+ 461us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:28:58.222279 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:28:58.222283 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:28:58.222871 (+ 588us) client_negotiation.cc:189] Negotiated authn=SASL
0806 06:28:58.223053 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:28:58.223058 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:28:58.223911 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:28:58.223912 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:28:58.224866 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:28:58.224868 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:28:58.225194 (+ 326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:28:58.225195 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:28:58.225298 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:28:58.225751 (+ 453us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 06:28:58.225760 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0806 06:28:58.227089 (+ 1329us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 06:28:58.228378 (+ 1289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:28:58.228384 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:28:58.228385 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:28:58.228580 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:28:58.228746 (+ 166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:28:58.228747 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:28:58.228747 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:28:58.228830 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:28:58.229003 (+ 173us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 06:28:58.229005 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 06:28:58.229103 (+ 98us) client_negotiation.cc:769] Sending connection context
0806 06:28:58.229177 (+ 74us) client_negotiation.cc:240] Negotiation successful
0806 06:28:58.229286 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":80,"threads_started":1}
I20260806 06:28:58.229908 10150 heartbeater.cc:347] Connected to a master server at 127.21.12.190:36809
I20260806 06:28:58.229995 10150 heartbeater.cc:464] Registering TS with master...
I20260806 06:28:58.230157 10150 heartbeater.cc:511] Master 127.21.12.190:36809 requested a full tablet report, sending...
I20260806 06:28:58.230538 9480 ts_manager.cc:194] Registered new tserver with Master: b778fbcc33a740b0876dd34c9fa6829c (127.21.12.131:32863)
I20260806 06:28:58.230865 9480 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.12.131@KRBTEST.COM'} at 127.21.12.131:33371
I20260806 06:28:58.234999 21554 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 06:28:58.241600 9541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:28:58.236259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39544 (local address 127.21.12.190:36809)
0806 06:28:58.236315 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0806 06:28:58.236317 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:28:58.236424 (+ 107us) server_negotiation.cc:407] Connection header received
0806 06:28:58.236488 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:28:58.236489 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:28:58.236499 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:28:58.236540 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:28:58.237080 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:28:58.237536 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:28:58.238235 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:28:58.238486 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:28:58.238540 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:28:58.239536 (+ 996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:28:58.239537 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:28:58.239538 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 06:28:58.239544 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:28:58.240591 (+ 1047us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:28:58.240883 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:28:58.240884 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:28:58.240884 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:28:58.240939 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:28:58.241102 (+ 163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:28:58.241103 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:28:58.241103 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:28:58.241243 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:28:58.241333 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0806 06:28:58.241435 (+ 102us) server_negotiation.cc:299] Negotiation successful
0806 06:28:58.241515 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260806 06:28:58.242882 9480 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39544:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 06:28:58.244503 9480 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 06:28:58.250648 10167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:28:58.246774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48942 (local address 127.21.12.131:32863)
0806 06:28:58.247088 (+ 314us) server_negotiation.cc:206] Beginning negotiation
0806 06:28:58.247093 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 06:28:58.247101 (+ 8us) server_negotiation.cc:407] Connection header received
0806 06:28:58.247131 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:28:58.247132 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:28:58.247143 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:28:58.247257 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 06:28:58.247386 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:28:58.248467 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:28:58.249535 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:28:58.250238 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:28:58.250330 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:28:58.250388 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0806 06:28:58.250533 (+ 145us) server_negotiation.cc:299] Negotiation successful
0806 06:28:58.250570 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":287,"thread_start_us":91,"threads_started":1}
I20260806 06:28:58.250860 10165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:28:58.246688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.131:32863 (local address 127.0.0.1:48942)
0806 06:28:58.246927 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0806 06:28:58.246939 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 06:28:58.246984 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:28:58.247275 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:28:58.247276 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:28:58.247278 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 06:28:58.247321 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:28:58.247322 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:28:58.248548 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:28:58.248549 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:28:58.249442 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:28:58.249445 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:28:58.250332 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:28:58.250334 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:28:58.250423 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:28:58.250438 (+ 15us) client_negotiation.cc:769] Sending connection context
0806 06:28:58.250513 (+ 75us) client_negotiation.cc:240] Negotiation successful
0806 06:28:58.250754 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227,"mutex_wait_us":2,"thread_start_us":40,"threads_started":1}
I20260806 06:28:58.250963 10163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:28:58.246498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.129:42271 (local address 127.0.0.1:42260)
0806 06:28:58.246605 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0806 06:28:58.246645 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0806 06:28:58.246708 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:28:58.247020 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:28:58.247034 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:28:58.247037 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 06:28:58.247104 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:28:58.247106 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:28:58.248398 (+ 1292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:28:58.248399 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:28:58.249878 (+ 1479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:28:58.249880 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:28:58.250761 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:28:58.250762 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:28:58.250829 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:28:58.250833 (+ 4us) client_negotiation.cc:769] Sending connection context
0806 06:28:58.250863 (+ 30us) client_negotiation.cc:240] Negotiation successful
0806 06:28:58.250904 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":80,"mutex_wait_us":70,"thread_start_us":47,"threads_started":1}
I20260806 06:28:58.251029 10168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:28:58.246925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51796 (local address 127.21.12.130:43341)
0806 06:28:58.247248 (+ 323us) server_negotiation.cc:206] Beginning negotiation
0806 06:28:58.247253 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 06:28:58.247263 (+ 10us) server_negotiation.cc:407] Connection header received
0806 06:28:58.247298 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:28:58.247300 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:28:58.247316 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:28:58.247386 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 06:28:58.247597 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:28:58.248307 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:28:58.249734 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:28:58.250496 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:28:58.250557 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:28:58.250608 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0806 06:28:58.250903 (+ 295us) server_negotiation.cc:299] Negotiation successful
0806 06:28:58.250942 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":288,"thread_start_us":80,"threads_started":1}
I20260806 06:28:58.251111 10166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:28:58.246803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.130:43341 (local address 127.0.0.1:51796)
0806 06:28:58.247095 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0806 06:28:58.247105 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0806 06:28:58.247147 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:28:58.247489 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:28:58.247490 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:28:58.247492 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 06:28:58.247526 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:28:58.247527 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:28:58.248409 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:28:58.248410 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:28:58.249661 (+ 1251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:28:58.249663 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:28:58.250640 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:28:58.250641 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:28:58.250705 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:28:58.250710 (+ 5us) client_negotiation.cc:769] Sending connection context
0806 06:28:58.250764 (+ 54us) client_negotiation.cc:240] Negotiation successful
0806 06:28:58.250807 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"mutex_wait_us":35,"thread_start_us":58,"threads_started":1}
I20260806 06:28:58.251276 10164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:28:58.246607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42260 (local address 127.21.12.129:42271)
0806 06:28:58.246833 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0806 06:28:58.246839 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 06:28:58.246850 (+ 11us) server_negotiation.cc:407] Connection header received
0806 06:28:58.246890 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:28:58.246892 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:28:58.246908 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:28:58.246999 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 06:28:58.247181 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:28:58.248288 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:28:58.249949 (+ 1661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:28:58.250658 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:28:58.251060 (+ 402us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:28:58.251130 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0806 06:28:58.251158 (+ 28us) server_negotiation.cc:299] Negotiation successful
0806 06:28:58.251201 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":189,"thread_start_us":67,"threads_started":1}
I20260806 06:28:58.251617 10053 tablet_service.cc:1467] Processing CreateTablet for tablet 3a80c47c38454f9a912330f3f42c76ba (DEFAULT_TABLE table=test-table [id=5d36c565a0df43a0a73bfb9292c6ac78]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 06:28:58.251968 10053 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a80c47c38454f9a912330f3f42c76ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 06:28:58.251899 9669 tablet_service.cc:1467] Processing CreateTablet for tablet 3a80c47c38454f9a912330f3f42c76ba (DEFAULT_TABLE table=test-table [id=5d36c565a0df43a0a73bfb9292c6ac78]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 06:28:58.251900 9856 tablet_service.cc:1467] Processing CreateTablet for tablet 3a80c47c38454f9a912330f3f42c76ba (DEFAULT_TABLE table=test-table [id=5d36c565a0df43a0a73bfb9292c6ac78]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 06:28:58.252236 9856 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a80c47c38454f9a912330f3f42c76ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 06:28:58.252236 9669 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a80c47c38454f9a912330f3f42c76ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 06:28:58.254376 10169 tablet_bootstrap.cc:492] T 3a80c47c38454f9a912330f3f42c76ba P b778fbcc33a740b0876dd34c9fa6829c: Bootstrap starting.
I20260806 06:28:58.254969 10169 tablet_bootstrap.cc:654] T 3a80c47c38454f9a912330f3f42c76ba P b778fbcc33a740b0876dd34c9fa6829c: Neither blocks nor log segments found. Creating new log.
I20260806 06:28:58.255391 10170 tablet_bootstrap.cc:492] T 3a80c47c38454f9a912330f3f42c76ba P e7a2421395c9473e91a329c11588a784: Bootstrap starting.
I20260806 06:28:58.255447 10169 log.cc:824] T 3a80c47c38454f9a912330f3f42c76ba P b778fbcc33a740b0876dd34c9fa6829c: Log is configured to *not* fsync() on all Append() calls
I20260806 06:28:58.255605 10172 tablet_bootstrap.cc:492] T 3a80c47c38454f9a912330f3f42c76ba P a397a32a5c4c4681951c1de266125f9c: Bootstrap starting.
I20260806 06:28:58.256145 10172 tablet_bootstrap.cc:654] T 3a80c47c38454f9a912330f3f42c76ba P a397a32a5c4c4681951c1de266125f9c: Neither blocks nor log segments found. Creating new log.
I20260806 06:28:58.256153 10169 tablet_bootstrap.cc:492] T 3a80c47c38454f9a912330f3f42c76ba P b778fbcc33a740b0876dd34c9fa6829c: No bootstrap required, opened a new log
I20260806 06:28:58.256210 10169 ts_tablet_manager.cc:1397] T 3a80c47c38454f9a912330f3f42c76ba P b778fbcc33a740b0876dd34c9fa6829c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260806 06:28:58.256594 10172 log.cc:824] T 3a80c47c38454f9a912330f3f42c76ba P a397a32a5c4c4681951c1de266125f9c: Log is configured to *not* fsync() on all Append() calls
I20260806 06:28:58.257298 10170 tablet_bootstrap.cc:654] T 3a80c47c38454f9a912330f3f42c76ba P e7a2421395c9473e91a329c11588a784: Neither blocks nor log segments found. Creating new log.
I20260806 06:28:58.257328 10172 tablet_bootstrap.cc:492] T 3a80c47c38454f9a912330f3f42c76ba P a397a32a5c4c4681951c1de266125f9c: No bootstrap required, opened a new log
I20260806 06:28:58.257385 10172 ts_tablet_manager.cc:1397] T 3a80c47c38454f9a912330f3f42c76ba P a397a32a5c4c4681951c1de266125f9c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260806 06:28:58.257757 10170 log.cc:824] T 3a80c47c38454f9a912330f3f42c76ba P e7a2421395c9473e91a329c11588a784: Log is configured to *not* fsync() on all Append() calls
I20260806 06:28:58.257818 10169 raft_consensus.cc:348] T 3a80c47c38454f9a912330f3f42c76ba P b778fbcc33a740b0876dd34c9fa6829c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7a2421395c9473e91a329c11588a784" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 42271 } } peers { permanent_uuid: "b778fbcc33a740b0876dd34c9fa6829c" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 32863 } } peers { permanent_uuid: "a397a32a5c4c4681951c1de266125f9c" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 43341 } }
I20260806 06:28:58.257984 10169 raft_consensus.cc:374] T 3a80c47c38454f9a912330f3f42c76ba P b778fbcc33a740b0876dd34c9fa6829c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 06:28:58.258031 10169 raft_consensus.cc:729] T 3a80c47c38454f9a912330f3f42c76ba P b778fbcc33a740b0876dd34c9fa6829c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b778fbcc33a740b0876dd34c9fa6829c, State: Initialized, Role: FOLLOWER
I20260806 06:28:58.258152 10169 consensus_queue.cc:260] T 3a80c47c38454f9a912330f3f42c76ba P b778fbcc33a740b0876dd34c9fa6829c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7a2421395c9473e91a329c11588a784" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 42271 } } peers { permanent_uuid: "b778fbcc33a740b0876dd34c9fa6829c" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 32863 } } peers { permanent_uuid: "a397a32a5c4c4681951c1de266125f9c" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 43341 } }
I20260806 06:28:58.258461 10169 ts_tablet_manager.cc:1428] T 3a80c47c38454f9a912330f3f42c76ba P b778fbcc33a740b0876dd34c9fa6829c: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20260806 06:28:58.258479 10170 tablet_bootstrap.cc:492] T 3a80c47c38454f9a912330f3f42c76ba P e7a2421395c9473e91a329c11588a784: No bootstrap required, opened a new log
I20260806 06:28:58.258538 10170 ts_tablet_manager.cc:1397] T 3a80c47c38454f9a912330f3f42c76ba P e7a2421395c9473e91a329c11588a784: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260806 06:28:58.258615 10150 heartbeater.cc:503] Master 127.21.12.190:36809 was elected leader, sending a full tablet report...
I20260806 06:28:58.259239 10172 raft_consensus.cc:348] T 3a80c47c38454f9a912330f3f42c76ba P a397a32a5c4c4681951c1de266125f9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7a2421395c9473e91a329c11588a784" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 42271 } } peers { permanent_uuid: "b778fbcc33a740b0876dd34c9fa6829c" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 32863 } } peers { permanent_uuid: "a397a32a5c4c4681951c1de266125f9c" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 43341 } }
I20260806 06:28:58.259397 10172 raft_consensus.cc:374] T 3a80c47c38454f9a912330f3f42c76ba P a397a32a5c4c4681951c1de266125f9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 06:28:58.259443 10172 raft_consensus.cc:729] T 3a80c47c38454f9a912330f3f42c76ba P a397a32a5c4c4681951c1de266125f9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a397a32a5c4c4681951c1de266125f9c, State: Initialized, Role: FOLLOWER
I20260806 06:28:58.259557 10172 consensus_queue.cc:260] T 3a80c47c38454f9a912330f3f42c76ba P a397a32a5c4c4681951c1de266125f9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7a2421395c9473e91a329c11588a784" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 42271 } } peers { permanent_uuid: "b778fbcc33a740b0876dd34c9fa6829c" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 32863 } } peers { permanent_uuid: "a397a32a5c4c4681951c1de266125f9c" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 43341 } }
I20260806 06:28:58.259788 10172 ts_tablet_manager.cc:1428] T 3a80c47c38454f9a912330f3f42c76ba P a397a32a5c4c4681951c1de266125f9c: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260806 06:28:58.259836 9957 heartbeater.cc:503] Master 127.21.12.190:36809 was elected leader, sending a full tablet report...
I20260806 06:28:58.260274 10170 raft_consensus.cc:348] T 3a80c47c38454f9a912330f3f42c76ba P e7a2421395c9473e91a329c11588a784 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7a2421395c9473e91a329c11588a784" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 42271 } } peers { permanent_uuid: "b778fbcc33a740b0876dd34c9fa6829c" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 32863 } } peers { permanent_uuid: "a397a32a5c4c4681951c1de266125f9c" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 43341 } }
I20260806 06:28:58.260394 10170 raft_consensus.cc:374] T 3a80c47c38454f9a912330f3f42c76ba P e7a2421395c9473e91a329c11588a784 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 06:28:58.260425 10170 raft_consensus.cc:729] T 3a80c47c38454f9a912330f3f42c76ba P e7a2421395c9473e91a329c11588a784 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7a2421395c9473e91a329c11588a784, State: Initialized, Role: FOLLOWER
I20260806 06:28:58.260529 10170 consensus_queue.cc:260] T 3a80c47c38454f9a912330f3f42c76ba P e7a2421395c9473e91a329c11588a784 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7a2421395c9473e91a329c11588a784" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 42271 } } peers { permanent_uuid: "b778fbcc33a740b0876dd34c9fa6829c" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 32863 } } peers { permanent_uuid: "a397a32a5c4c4681951c1de266125f9c" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 43341 } }
I20260806 06:28:58.260797 10170 ts_tablet_manager.cc:1428] T 3a80c47c38454f9a912330f3f42c76ba P e7a2421395c9473e91a329c11588a784: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260806 06:28:58.260823 9763 heartbeater.cc:503] Master 127.21.12.190:36809 was elected leader, sending a full tablet report...
I20260806 06:28:58.310635 10177 raft_consensus.cc:482] T 3a80c47c38454f9a912330f3f42c76ba P b778fbcc33a740b0876dd34c9fa6829c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 06:28:58.310781 10177 raft_consensus.cc:504] T 3a80c47c38454f9a912330f3f42c76ba P b778fbcc33a740b0876dd34c9fa6829c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7a2421395c9473e91a329c11588a784" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 42271 } } peers { permanent_uuid: "b778fbcc33a740b0876dd34c9fa6829c" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 32863 } } peers { permanent_uuid: "a397a32a5c4c4681951c1de266125f9c" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 43341 } }
I20260806 06:28:58.311200 10177 leader_election.cc:288] T 3a80c47c38454f9a912330f3f42c76ba P b778fbcc33a740b0876dd34c9fa6829c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e7a2421395c9473e91a329c11588a784 (127.21.12.129:42271), a397a32a5c4c4681951c1de266125f9c (127.21.12.130:43341)
I20260806 06:28:58.313822 10164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:28:58.311430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.131:33311 (local address 127.21.12.129:42271)
0806 06:28:58.311476 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0806 06:28:58.311477 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0806 06:28:58.311561 (+ 84us) server_negotiation.cc:407] Connection header received
0806 06:28:58.311591 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:28:58.311592 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:28:58.311604 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:28:58.311651 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 06:28:58.311801 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:28:58.312261 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:28:58.313133 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:28:58.313510 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:28:58.313571 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:28:58.313580 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0806 06:28:58.313723 (+ 143us) server_negotiation.cc:299] Negotiation successful
0806 06:28:58.313759 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260806 06:28:58.313822 10154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:28:58.311351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.129:42271 (local address 127.21.12.131:33311)
0806 06:28:58.311408 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0806 06:28:58.311421 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 06:28:58.311473 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:28:58.311670 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:28:58.311671 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:28:58.311672 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 06:28:58.311742 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:28:58.311744 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:28:58.312335 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:28:58.312336 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:28:58.313046 (+ 710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:28:58.313047 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:28:58.313581 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:28:58.313582 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:28:58.313657 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:28:58.313660 (+ 3us) client_negotiation.cc:769] Sending connection context
0806 06:28:58.313707 (+ 47us) client_negotiation.cc:240] Negotiation successful
0806 06:28:58.313749 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260806 06:28:58.313962 10180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:28:58.311429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.130:43341 (local address 127.21.12.131:52161)
0806 06:28:58.311617 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0806 06:28:58.311625 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0806 06:28:58.311683 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:28:58.311807 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:28:58.311808 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:28:58.311809 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 06:28:58.311882 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:28:58.311883 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:28:58.312465 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:28:58.312466 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:28:58.313191 (+ 725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:28:58.313192 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:28:58.313737 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:28:58.313738 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:28:58.313799 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:28:58.313803 (+ 4us) client_negotiation.cc:769] Sending connection context
0806 06:28:58.313848 (+ 45us) client_negotiation.cc:240] Negotiation successful
0806 06:28:58.313883 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":123,"threads_started":1}
I20260806 06:28:58.313962 10168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:28:58.311525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.131:52161 (local address 127.21.12.130:43341)
0806 06:28:58.311569 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0806 06:28:58.311571 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:28:58.311688 (+ 117us) server_negotiation.cc:407] Connection header received
0806 06:28:58.311741 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:28:58.311742 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:28:58.311752 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:28:58.311800 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 06:28:58.311943 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:28:58.312392 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:28:58.313277 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:28:58.313662 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:28:58.313725 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:28:58.313732 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0806 06:28:58.313862 (+ 130us) server_negotiation.cc:299] Negotiation successful
0806 06:28:58.313901 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260806 06:28:58.314381 9700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a80c47c38454f9a912330f3f42c76ba" candidate_uuid: "b778fbcc33a740b0876dd34c9fa6829c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7a2421395c9473e91a329c11588a784" is_pre_election: true
I20260806 06:28:58.314381 9884 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a80c47c38454f9a912330f3f42c76ba" candidate_uuid: "b778fbcc33a740b0876dd34c9fa6829c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a397a32a5c4c4681951c1de266125f9c" is_pre_election: true
I20260806 06:28:58.314554 9700 raft_consensus.cc:2454] T 3a80c47c38454f9a912330f3f42c76ba P e7a2421395c9473e91a329c11588a784 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b778fbcc33a740b0876dd34c9fa6829c in term 0.
I20260806 06:28:58.314553 9884 raft_consensus.cc:2454] T 3a80c47c38454f9a912330f3f42c76ba P a397a32a5c4c4681951c1de266125f9c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b778fbcc33a740b0876dd34c9fa6829c in term 0.
I20260806 06:28:58.314795 9988 leader_election.cc:302] T 3a80c47c38454f9a912330f3f42c76ba P b778fbcc33a740b0876dd34c9fa6829c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a397a32a5c4c4681951c1de266125f9c, b778fbcc33a740b0876dd34c9fa6829c; no voters:
I20260806 06:28:58.314913 10177 raft_consensus.cc:2781] T 3a80c47c38454f9a912330f3f42c76ba P b778fbcc33a740b0876dd34c9fa6829c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 06:28:58.314971 10177 raft_consensus.cc:482] T 3a80c47c38454f9a912330f3f42c76ba P b778fbcc33a740b0876dd34c9fa6829c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 06:28:58.314999 10177 raft_consensus.cc:3037] T 3a80c47c38454f9a912330f3f42c76ba P b778fbcc33a740b0876dd34c9fa6829c [term 0 FOLLOWER]: Advancing to term 1
I20260806 06:28:58.315676 10177 raft_consensus.cc:504] T 3a80c47c38454f9a912330f3f42c76ba P b778fbcc33a740b0876dd34c9fa6829c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7a2421395c9473e91a329c11588a784" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 42271 } } peers { permanent_uuid: "b778fbcc33a740b0876dd34c9fa6829c" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 32863 } } peers { permanent_uuid: "a397a32a5c4c4681951c1de266125f9c" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 43341 } }
I20260806 06:28:58.315783 10177 leader_election.cc:288] T 3a80c47c38454f9a912330f3f42c76ba P b778fbcc33a740b0876dd34c9fa6829c [CANDIDATE]: Term 1 election: Requested vote from peers e7a2421395c9473e91a329c11588a784 (127.21.12.129:42271), a397a32a5c4c4681951c1de266125f9c (127.21.12.130:43341)
I20260806 06:28:58.315943 9700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a80c47c38454f9a912330f3f42c76ba" candidate_uuid: "b778fbcc33a740b0876dd34c9fa6829c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7a2421395c9473e91a329c11588a784"
I20260806 06:28:58.315979 9884 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a80c47c38454f9a912330f3f42c76ba" candidate_uuid: "b778fbcc33a740b0876dd34c9fa6829c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a397a32a5c4c4681951c1de266125f9c"
I20260806 06:28:58.316039 9700 raft_consensus.cc:3037] T 3a80c47c38454f9a912330f3f42c76ba P e7a2421395c9473e91a329c11588a784 [term 0 FOLLOWER]: Advancing to term 1
I20260806 06:28:58.316049 9884 raft_consensus.cc:3037] T 3a80c47c38454f9a912330f3f42c76ba P a397a32a5c4c4681951c1de266125f9c [term 0 FOLLOWER]: Advancing to term 1
I20260806 06:28:58.316910 9884 raft_consensus.cc:2454] T 3a80c47c38454f9a912330f3f42c76ba P a397a32a5c4c4681951c1de266125f9c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b778fbcc33a740b0876dd34c9fa6829c in term 1.
I20260806 06:28:58.316910 9700 raft_consensus.cc:2454] T 3a80c47c38454f9a912330f3f42c76ba P e7a2421395c9473e91a329c11588a784 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b778fbcc33a740b0876dd34c9fa6829c in term 1.
I20260806 06:28:58.317090 9988 leader_election.cc:302] T 3a80c47c38454f9a912330f3f42c76ba P b778fbcc33a740b0876dd34c9fa6829c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b778fbcc33a740b0876dd34c9fa6829c, e7a2421395c9473e91a329c11588a784; no voters:
I20260806 06:28:58.317198 10177 raft_consensus.cc:2781] T 3a80c47c38454f9a912330f3f42c76ba P b778fbcc33a740b0876dd34c9fa6829c [term 1 FOLLOWER]: Leader election won for term 1
I20260806 06:28:58.317342 10177 raft_consensus.cc:686] T 3a80c47c38454f9a912330f3f42c76ba P b778fbcc33a740b0876dd34c9fa6829c [term 1 LEADER]: Becoming Leader. State: Replica: b778fbcc33a740b0876dd34c9fa6829c, State: Running, Role: LEADER
I20260806 06:28:58.317430 10177 consensus_queue.cc:237] T 3a80c47c38454f9a912330f3f42c76ba P b778fbcc33a740b0876dd34c9fa6829c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7a2421395c9473e91a329c11588a784" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 42271 } } peers { permanent_uuid: "b778fbcc33a740b0876dd34c9fa6829c" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 32863 } } peers { permanent_uuid: "a397a32a5c4c4681951c1de266125f9c" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 43341 } }
I20260806 06:28:58.318236 9479 catalog_manager.cc:5495] T 3a80c47c38454f9a912330f3f42c76ba P b778fbcc33a740b0876dd34c9fa6829c reported cstate change: term changed from 0 to 1, leader changed from <none> to b778fbcc33a740b0876dd34c9fa6829c (127.21.12.131). New cstate: current_term: 1 leader_uuid: "b778fbcc33a740b0876dd34c9fa6829c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7a2421395c9473e91a329c11588a784" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 42271 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b778fbcc33a740b0876dd34c9fa6829c" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 32863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a397a32a5c4c4681951c1de266125f9c" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 43341 } health_report { overall_health: UNKNOWN } } }
W20260806 06:28:58.319664 9958 tablet.cc:2369] T 3a80c47c38454f9a912330f3f42c76ba P a397a32a5c4c4681951c1de266125f9c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 06:28:58.326061 10167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:28:58.324097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48944 (local address 127.21.12.131:32863)
0806 06:28:58.324133 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0806 06:28:58.324134 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0806 06:28:58.324141 (+ 7us) server_negotiation.cc:407] Connection header received
0806 06:28:58.324196 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:28:58.324196 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:28:58.324210 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:28:58.324270 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 06:28:58.324345 (+ 75us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:28:58.324785 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:28:58.325500 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:28:58.325667 (+ 167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:28:58.325714 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:28:58.325833 (+ 119us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 06:28:58.325920 (+ 87us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 06:28:58.325970 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0806 06:28:58.325984 (+ 14us) server_negotiation.cc:299] Negotiation successful
0806 06:28:58.325997 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260806 06:28:58.327689 9884 raft_consensus.cc:1261] T 3a80c47c38454f9a912330f3f42c76ba P a397a32a5c4c4681951c1de266125f9c [term 1 FOLLOWER]: Refusing update from remote peer b778fbcc33a740b0876dd34c9fa6829c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 06:28:58.327697 9700 raft_consensus.cc:1261] T 3a80c47c38454f9a912330f3f42c76ba P e7a2421395c9473e91a329c11588a784 [term 1 FOLLOWER]: Refusing update from remote peer b778fbcc33a740b0876dd34c9fa6829c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 06:28:58.328001 10177 consensus_queue.cc:1035] T 3a80c47c38454f9a912330f3f42c76ba P b778fbcc33a740b0876dd34c9fa6829c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7a2421395c9473e91a329c11588a784" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 42271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 06:28:58.328121 10181 consensus_queue.cc:1035] T 3a80c47c38454f9a912330f3f42c76ba P b778fbcc33a740b0876dd34c9fa6829c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a397a32a5c4c4681951c1de266125f9c" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 43341 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 06:28:58.330406 10184 mvcc.cc:205] Tried to move back new op lower bound from 7315446736188010496 to 7315446736148869120. Current Snapshot: MvccSnapshot[applied={T|T < 7315446736188010496}]
I20260806 06:28:58.330487 10186 mvcc.cc:205] Tried to move back new op lower bound from 7315446736188010496 to 7315446736148869120. Current Snapshot: MvccSnapshot[applied={T|T < 7315446736188010496}]
I20260806 06:28:58.330775 10187 mvcc.cc:205] Tried to move back new op lower bound from 7315446736188010496 to 7315446736148869120. Current Snapshot: MvccSnapshot[applied={T|T < 7315446736188010496}]
I20260806 06:28:58.332356 9480 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39544:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 06:28:58.332445 9480 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39544:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 06:28:58.333184 9480 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2ad50aad27464833af0c85627af4d871: Sending DeleteTablet for 3 replicas of tablet 3a80c47c38454f9a912330f3f42c76ba
I20260806 06:28:58.333500 21554 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHhLhT/build/release/bin/kudu with pid 9575
I20260806 06:28:58.333482 9669 tablet_service.cc:1514] Processing DeleteTablet for tablet 3a80c47c38454f9a912330f3f42c76ba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 06:28:58 UTC) from {username='kudu', principal='kudu/127.21.12.190@KRBTEST.COM'} at 127.0.0.1:42260
I20260806 06:28:58.333519 10053 tablet_service.cc:1514] Processing DeleteTablet for tablet 3a80c47c38454f9a912330f3f42c76ba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 06:28:58 UTC) from {username='kudu', principal='kudu/127.21.12.190@KRBTEST.COM'} at 127.0.0.1:48942
I20260806 06:28:58.334116 9856 tablet_service.cc:1514] Processing DeleteTablet for tablet 3a80c47c38454f9a912330f3f42c76ba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 06:28:58 UTC) from {username='kudu', principal='kudu/127.21.12.190@KRBTEST.COM'} at 127.0.0.1:51796
I20260806 06:28:58.341426 10202 tablet_replica.cc:330] T 3a80c47c38454f9a912330f3f42c76ba P a397a32a5c4c4681951c1de266125f9c: stopping tablet replica
I20260806 06:28:58.341566 10202 raft_consensus.cc:2229] T 3a80c47c38454f9a912330f3f42c76ba P a397a32a5c4c4681951c1de266125f9c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 06:28:58.341683 10202 raft_consensus.cc:2258] T 3a80c47c38454f9a912330f3f42c76ba P a397a32a5c4c4681951c1de266125f9c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 06:28:58.342114 10202 ts_tablet_manager.cc:1905] T 3a80c47c38454f9a912330f3f42c76ba P a397a32a5c4c4681951c1de266125f9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 06:28:58.342409 10201 tablet_replica.cc:330] T 3a80c47c38454f9a912330f3f42c76ba P b778fbcc33a740b0876dd34c9fa6829c: stopping tablet replica
I20260806 06:28:58.342564 10201 raft_consensus.cc:2229] T 3a80c47c38454f9a912330f3f42c76ba P b778fbcc33a740b0876dd34c9fa6829c [term 1 LEADER]: Raft consensus shutting down.
I20260806 06:28:58.342717 10201 raft_consensus.cc:2258] T 3a80c47c38454f9a912330f3f42c76ba P b778fbcc33a740b0876dd34c9fa6829c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 06:28:58.343158 10201 ts_tablet_manager.cc:1905] T 3a80c47c38454f9a912330f3f42c76ba P b778fbcc33a740b0876dd34c9fa6829c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 06:28:58.344556 10202 ts_tablet_manager.cc:1918] T 3a80c47c38454f9a912330f3f42c76ba P a397a32a5c4c4681951c1de266125f9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 06:28:58.344640 10202 log.cc:1196] T 3a80c47c38454f9a912330f3f42c76ba P a397a32a5c4c4681951c1de266125f9c: Deleting WAL directory at /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/minicluster-data/ts-1/wal/wals/3a80c47c38454f9a912330f3f42c76ba
I20260806 06:28:58.344770 10201 ts_tablet_manager.cc:1918] T 3a80c47c38454f9a912330f3f42c76ba P b778fbcc33a740b0876dd34c9fa6829c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 06:28:58.344839 10201 log.cc:1196] T 3a80c47c38454f9a912330f3f42c76ba P b778fbcc33a740b0876dd34c9fa6829c: Deleting WAL directory at /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/minicluster-data/ts-2/wal/wals/3a80c47c38454f9a912330f3f42c76ba
I20260806 06:28:58.344924 10202 ts_tablet_manager.cc:1939] T 3a80c47c38454f9a912330f3f42c76ba P a397a32a5c4c4681951c1de266125f9c: Deleting consensus metadata
I20260806 06:28:58.345093 10201 ts_tablet_manager.cc:1939] T 3a80c47c38454f9a912330f3f42c76ba P b778fbcc33a740b0876dd34c9fa6829c: Deleting consensus metadata
I20260806 06:28:58.345420 9459 catalog_manager.cc:4841] TS a397a32a5c4c4681951c1de266125f9c (127.21.12.130:43341): tablet 3a80c47c38454f9a912330f3f42c76ba (table test-table [id=5d36c565a0df43a0a73bfb9292c6ac78]) successfully deleted
I20260806 06:28:58.345549 9459 catalog_manager.cc:4841] TS b778fbcc33a740b0876dd34c9fa6829c (127.21.12.131:32863): tablet 3a80c47c38454f9a912330f3f42c76ba (table test-table [id=5d36c565a0df43a0a73bfb9292c6ac78]) successfully deleted
W20260806 06:28:58.348954 9459 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.12.129:42271 (error 108)
I20260806 06:28:58.349193 21554 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHhLhT/build/release/bin/kudu with pid 9771
I20260806 06:28:58.351621 10166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:28:58.349335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.129:42271 (local address 127.0.0.1:42268)
0806 06:28:58.351535 (+ 2200us) negotiation.cc:107] Waiting for socket to connect
0806 06:28:58.351585 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.129:42271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2171}
W20260806 06:28:58.352653 9459 catalog_manager.cc:4584] TS e7a2421395c9473e91a329c11588a784 (127.21.12.129:42271): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3a80c47c38454f9a912330f3f42c76ba: Network error: Client connection negotiation failed: client connection to 127.21.12.129:42271: connect: Connection refused (error 111)
I20260806 06:28:58.363075 21554 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHhLhT/build/release/bin/kudu with pid 9963
I20260806 06:28:58.372069 21554 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHhLhT/build/release/bin/kudu with pid 9436
I20260806 06:28:58.380787 21554 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHhLhT/build/release/bin/kudu
/tmp/dist-test-taskFHhLhT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.12.190:36809
--webserver_interface=127.21.12.190
--webserver_port=44209
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.190.keytab
--principal=kudu/127.21.12.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.148:41671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.12.190:36809
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 06:28:58.461005 10204 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 06:28:58.461210 10204 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 06:28:58.461230 10204 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 06:28:58.462388 10204 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 06:28:58.462426 10204 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 06:28:58.462440 10204 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 06:28:58.462458 10204 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 06:28:58.462481 10204 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 06:28:58.463958 10204 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.148:41671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.21.12.190:36809
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.190.keytab
--principal=kudu/127.21.12.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.12.190:36809
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.12.190
--webserver_port=44209
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10204
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 05:11:02 UTC on ccycloud.quasar-hvzbbz.root.comops.site
build id 4569617
I20260806 06:28:58.464182 10204 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 06:28:58.464381 10204 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 06:28:58.474105 10218 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:28:58.474073 10213 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:28:58.474093 10212 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:28:58.474261 10204 server_base.cc:1034] running on GCE node
I20260806 06:28:58.474512 10204 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 06:28:58.474822 10204 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 06:28:58.475953 10204 hybrid_clock.cc:648] HybridClock initialized: now 1785997738475935 us; error 37 us; skew 500 ppm
Aug 06 06:28:58 dist-test-slave-4tdj krb5kdc[9417](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997738, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 06:28:58.478333 10204 init.cc:377] Logged in from keytab as kudu/127.21.12.190@KRBTEST.COM (short username kudu)
I20260806 06:28:58.478765 10204 webserver.cc:460] Webserver started at http://127.21.12.190:44209/ using document root <none> and password file <none>
I20260806 06:28:58.478973 10204 fs_manager.cc:359] Metadata directory not provided
I20260806 06:28:58.479032 10204 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 06:28:58.480311 10204 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260806 06:28:58.480916 10226 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 06:28:58.481076 10204 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260806 06:28:58.481163 10204 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/minicluster-data/master-0/data,/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/minicluster-data/master-0/wal
uuid: "2ad50aad27464833af0c85627af4d871"
format_stamp: "Formatted at 2026-08-06 06:28:57 on dist-test-slave-4tdj"
server_key: "e7f5ee43b1a59a3d790275abcf9c16b2"
server_key_iv: "4be3aafe4a3837f23cfd543aa9b2c504"
server_key_version: "encryptionkey@0"
I20260806 06:28:58.481534 10204 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 06:28:58.492527 10204 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 06:28:58.494546 10204 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 06:28:58.494670 10204 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 06:28:58.498426 10204 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.190:36809
I20260806 06:28:58.498468 10320 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.190:36809 every 8 connection(s)
I20260806 06:28:58.498826 10204 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/minicluster-data/master-0/data/info.pb
I20260806 06:28:58.500442 10321 sys_catalog.cc:263] Verifying existing consensus state
I20260806 06:28:58.501057 10321 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ad50aad27464833af0c85627af4d871: Bootstrap starting.
I20260806 06:28:58.505324 21554 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHhLhT/build/release/bin/kudu as pid 10204
I20260806 06:28:58.505648 21554 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHhLhT/build/release/bin/kudu
/tmp/dist-test-taskFHhLhT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.12.129:42271
--local_ip_for_outbound_sockets=127.21.12.129
--tserver_master_addrs=127.21.12.190:36809
--webserver_port=38883
--webserver_interface=127.21.12.129
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.129.keytab
--principal=kudu/127.21.12.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.148:41671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 06:28:58.509820 10321 log.cc:824] T 00000000000000000000000000000000 P 2ad50aad27464833af0c85627af4d871: Log is configured to *not* fsync() on all Append() calls
I20260806 06:28:58.511667 10321 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ad50aad27464833af0c85627af4d871: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 06:28:58.511937 10321 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ad50aad27464833af0c85627af4d871: Bootstrap complete.
I20260806 06:28:58.514060 10321 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2ad50aad27464833af0c85627af4d871 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ad50aad27464833af0c85627af4d871" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 36809 } }
I20260806 06:28:58.514281 10321 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2ad50aad27464833af0c85627af4d871 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ad50aad27464833af0c85627af4d871, State: Initialized, Role: FOLLOWER
I20260806 06:28:58.514469 10321 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2ad50aad27464833af0c85627af4d871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ad50aad27464833af0c85627af4d871" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 36809 } }
I20260806 06:28:58.514534 10321 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2ad50aad27464833af0c85627af4d871 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 06:28:58.514581 10321 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2ad50aad27464833af0c85627af4d871 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 06:28:58.514629 10321 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2ad50aad27464833af0c85627af4d871 [term 1 FOLLOWER]: Advancing to term 2
I20260806 06:28:58.515612 10321 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2ad50aad27464833af0c85627af4d871 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ad50aad27464833af0c85627af4d871" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 36809 } }
I20260806 06:28:58.515702 10321 leader_election.cc:302] T 00000000000000000000000000000000 P 2ad50aad27464833af0c85627af4d871 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2ad50aad27464833af0c85627af4d871; no voters:
I20260806 06:28:58.515851 10321 leader_election.cc:288] T 00000000000000000000000000000000 P 2ad50aad27464833af0c85627af4d871 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260806 06:28:58.515900 10328 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2ad50aad27464833af0c85627af4d871 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 06:28:58.516021 10328 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2ad50aad27464833af0c85627af4d871 [term 2 LEADER]: Becoming Leader. State: Replica: 2ad50aad27464833af0c85627af4d871, State: Running, Role: LEADER
I20260806 06:28:58.516127 10321 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2ad50aad27464833af0c85627af4d871 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 06:28:58.516115 10328 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2ad50aad27464833af0c85627af4d871 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ad50aad27464833af0c85627af4d871" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 36809 } }
I20260806 06:28:58.516462 10330 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ad50aad27464833af0c85627af4d871 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "2ad50aad27464833af0c85627af4d871" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ad50aad27464833af0c85627af4d871" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 36809 } } }
I20260806 06:28:58.516476 10329 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ad50aad27464833af0c85627af4d871 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2ad50aad27464833af0c85627af4d871. Latest consensus state: current_term: 2 leader_uuid: "2ad50aad27464833af0c85627af4d871" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ad50aad27464833af0c85627af4d871" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 36809 } } }
I20260806 06:28:58.516623 10330 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ad50aad27464833af0c85627af4d871 [sys.catalog]: This master's current role is: LEADER
I20260806 06:28:58.516641 10329 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ad50aad27464833af0c85627af4d871 [sys.catalog]: This master's current role is: LEADER
I20260806 06:28:58.516988 10340 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 06:28:58.517781 10340 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 06:28:58.518115 10340 catalog_manager.cc:1259] Loaded cluster ID: 377b98b622474c4ab577dd247b3d558d
I20260806 06:28:58.518152 10340 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 06:28:58.518399 10340 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260806 06:28:58.523339 10340 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260806 06:28:58.525934 10340 catalog_manager.cc:1504] Loading token signing keys...
I20260806 06:28:58.526110 10340 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 2ad50aad27464833af0c85627af4d871: Loaded TSK: 0
I20260806 06:28:58.526333 10340 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260806 06:28:58.590459 10324 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 06:28:58.590657 10324 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 06:28:58.590680 10324 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 06:28:58.591813 10324 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 06:28:58.591854 10324 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 06:28:58.591879 10324 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.12.129
I20260806 06:28:58.593333 10324 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.148:41671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.129.keytab
--principal=kudu/127.21.12.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.12.129:42271
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.12.129
--webserver_port=38883
--tserver_master_addrs=127.21.12.190:36809
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10324
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.12.129
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 05:11:02 UTC on ccycloud.quasar-hvzbbz.root.comops.site
build id 4569617
I20260806 06:28:58.593570 10324 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 06:28:58.593798 10324 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 06:28:58.599821 10357 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:28:58.599852 10358 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:28:58.599856 10361 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:28:58.600116 10324 server_base.cc:1034] running on GCE node
I20260806 06:28:58.600318 10324 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 06:28:58.600529 10324 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 06:28:58.601663 10324 hybrid_clock.cc:648] HybridClock initialized: now 1785997738601651 us; error 32 us; skew 500 ppm
Aug 06 06:28:58 dist-test-slave-4tdj krb5kdc[9417](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997738, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 06:28:58.604027 10324 init.cc:377] Logged in from keytab as kudu/127.21.12.129@KRBTEST.COM (short username kudu)
I20260806 06:28:58.604449 10324 webserver.cc:460] Webserver started at http://127.21.12.129:38883/ using document root <none> and password file <none>
I20260806 06:28:58.604758 10324 fs_manager.cc:359] Metadata directory not provided
I20260806 06:28:58.604818 10324 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 06:28:58.606151 10324 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260806 06:28:58.606843 10372 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 06:28:58.606983 10324 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260806 06:28:58.607052 10324 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/minicluster-data/ts-0/wal
uuid: "e7a2421395c9473e91a329c11588a784"
format_stamp: "Formatted at 2026-08-06 06:28:57 on dist-test-slave-4tdj"
server_key: "e01ae4fd34e538a96a7c18bb93f98cb4"
server_key_iv: "8d4919674b8d1227fbff0693830034c0"
server_key_version: "encryptionkey@0"
I20260806 06:28:58.607389 10324 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 06:28:58.624473 10324 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 06:28:58.626554 10324 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 06:28:58.626679 10324 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 06:28:58.626890 10324 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 06:28:58.627343 10384 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260806 06:28:58.628706 10324 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260806 06:28:58.628757 10324 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260806 06:28:58.628782 10324 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260806 06:28:58.629415 10324 ts_tablet_manager.cc:610] Registered 1 tablets
I20260806 06:28:58.629458 10324 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260806 06:28:58.629520 10384 tablet_bootstrap.cc:492] T 3a80c47c38454f9a912330f3f42c76ba P e7a2421395c9473e91a329c11588a784: Bootstrap starting.
I20260806 06:28:58.636699 10324 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.129:42271
I20260806 06:28:58.637187 10324 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/minicluster-data/ts-0/data/info.pb
I20260806 06:28:58.638525 10543 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.129:42271 every 8 connection(s)
I20260806 06:28:58.640268 10384 log.cc:824] T 3a80c47c38454f9a912330f3f42c76ba P e7a2421395c9473e91a329c11588a784: Log is configured to *not* fsync() on all Append() calls
I20260806 06:28:58.641491 21554 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHhLhT/build/release/bin/kudu as pid 10324
I20260806 06:28:58.641774 21554 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHhLhT/build/release/bin/kudu
/tmp/dist-test-taskFHhLhT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.12.130:43341
--local_ip_for_outbound_sockets=127.21.12.130
--tserver_master_addrs=127.21.12.190:36809
--webserver_port=43723
--webserver_interface=127.21.12.130
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.130.keytab
--principal=kudu/127.21.12.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.148:41671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 06:28:58.642655 10384 tablet_bootstrap.cc:492] T 3a80c47c38454f9a912330f3f42c76ba P e7a2421395c9473e91a329c11588a784: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 06:28:58.642900 10384 tablet_bootstrap.cc:492] T 3a80c47c38454f9a912330f3f42c76ba P e7a2421395c9473e91a329c11588a784: Bootstrap complete.
I20260806 06:28:58.643841 10384 ts_tablet_manager.cc:1397] T 3a80c47c38454f9a912330f3f42c76ba P e7a2421395c9473e91a329c11588a784: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.011s
Aug 06 06:28:58 dist-test-slave-4tdj krb5kdc[9417](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997738, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.129@KRBTEST.COM for kudu/127.21.12.190@KRBTEST.COM
I20260806 06:28:58.645308 10384 raft_consensus.cc:348] T 3a80c47c38454f9a912330f3f42c76ba P e7a2421395c9473e91a329c11588a784 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7a2421395c9473e91a329c11588a784" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 42271 } } peers { permanent_uuid: "b778fbcc33a740b0876dd34c9fa6829c" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 32863 } } peers { permanent_uuid: "a397a32a5c4c4681951c1de266125f9c" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 43341 } }
I20260806 06:28:58.645627 10384 raft_consensus.cc:729] T 3a80c47c38454f9a912330f3f42c76ba P e7a2421395c9473e91a329c11588a784 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7a2421395c9473e91a329c11588a784, State: Initialized, Role: FOLLOWER
I20260806 06:28:58.645771 10384 consensus_queue.cc:260] T 3a80c47c38454f9a912330f3f42c76ba P e7a2421395c9473e91a329c11588a784 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7a2421395c9473e91a329c11588a784" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 42271 } } peers { permanent_uuid: "b778fbcc33a740b0876dd34c9fa6829c" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 32863 } } peers { permanent_uuid: "a397a32a5c4c4681951c1de266125f9c" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 43341 } }
I20260806 06:28:58.646020 10384 ts_tablet_manager.cc:1428] T 3a80c47c38454f9a912330f3f42c76ba P e7a2421395c9473e91a329c11588a784: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20260806 06:28:58.649307 10549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:28:58.638035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.129:42143 (local address 127.21.12.190:36809)
0806 06:28:58.638225 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0806 06:28:58.638234 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 06:28:58.638522 (+ 288us) server_negotiation.cc:407] Connection header received
0806 06:28:58.639342 (+ 820us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:28:58.639354 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:28:58.639424 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:28:58.639588 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:28:58.640363 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:28:58.641599 (+ 1236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:28:58.643003 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:28:58.643565 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:28:58.643667 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:28:58.646075 (+ 2408us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:28:58.646079 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:28:58.646079 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 06:28:58.646091 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:28:58.647592 (+ 1501us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:28:58.648064 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:28:58.648066 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:28:58.648066 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:28:58.648144 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:28:58.648393 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:28:58.648393 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:28:58.648394 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:28:58.648580 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:28:58.648636 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0806 06:28:58.649053 (+ 417us) server_negotiation.cc:299] Negotiation successful
0806 06:28:58.649155 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":89,"threads_started":1}
I20260806 06:28:58.649509 10547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:28:58.637958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:36809 (local address 127.21.12.129:42143)
0806 06:28:58.638374 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0806 06:28:58.638419 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0806 06:28:58.639094 (+ 675us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:28:58.639630 (+ 536us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:28:58.639635 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:28:58.639969 (+ 334us) client_negotiation.cc:189] Negotiated authn=SASL
0806 06:28:58.640228 (+ 259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:28:58.640235 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:28:58.641724 (+ 1489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:28:58.641725 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:28:58.642884 (+ 1159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:28:58.642887 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:28:58.643667 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:28:58.643668 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:28:58.643791 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:28:58.644297 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 06:28:58.644306 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0806 06:28:58.645963 (+ 1657us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 06:28:58.647669 (+ 1706us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:28:58.647676 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:28:58.647679 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:28:58.647975 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:28:58.648201 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:28:58.648202 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:28:58.648202 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:28:58.648336 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:28:58.648692 (+ 356us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 06:28:58.648700 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 06:28:58.648893 (+ 193us) client_negotiation.cc:769] Sending connection context
0806 06:28:58.649023 (+ 130us) client_negotiation.cc:240] Negotiation successful
0806 06:28:58.649199 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":112,"threads_started":1}
I20260806 06:28:58.650189 10544 heartbeater.cc:347] Connected to a master server at 127.21.12.190:36809
I20260806 06:28:58.650279 10544 heartbeater.cc:464] Registering TS with master...
I20260806 06:28:58.650463 10544 heartbeater.cc:511] Master 127.21.12.190:36809 requested a full tablet report, sending...
I20260806 06:28:58.651214 10262 ts_manager.cc:194] Registered new tserver with Master: e7a2421395c9473e91a329c11588a784 (127.21.12.129:42271)
I20260806 06:28:58.652419 10262 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.12.129@KRBTEST.COM'} at 127.21.12.129:42143
I20260806 06:28:58.653250 10544 heartbeater.cc:503] Master 127.21.12.190:36809 was elected leader, sending a full tablet report...
I20260806 06:28:58.656428 10555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:28:58.651889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42276 (local address 127.21.12.129:42271)
0806 06:28:58.652981 (+ 1092us) server_negotiation.cc:206] Beginning negotiation
0806 06:28:58.652986 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 06:28:58.652996 (+ 10us) server_negotiation.cc:407] Connection header received
0806 06:28:58.653027 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:28:58.653029 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:28:58.653154 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:28:58.653230 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 06:28:58.653495 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:28:58.654150 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:28:58.655675 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:28:58.656108 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:28:58.656169 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:28:58.656209 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0806 06:28:58.656350 (+ 141us) server_negotiation.cc:299] Negotiation successful
0806 06:28:58.656382 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1062,"thread_start_us":67,"threads_started":1}
I20260806 06:28:58.656459 10554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:28:58.651827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.129:42271 (local address 127.0.0.1:42276)
0806 06:28:58.652754 (+ 927us) negotiation.cc:107] Waiting for socket to connect
0806 06:28:58.652789 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0806 06:28:58.652856 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:28:58.653304 (+ 448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:28:58.653308 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:28:58.653310 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 06:28:58.653374 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:28:58.653376 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:28:58.654234 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:28:58.654235 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:28:58.655594 (+ 1359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:28:58.655598 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:28:58.656177 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:28:58.656178 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:28:58.656255 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:28:58.656268 (+ 13us) client_negotiation.cc:769] Sending connection context
0806 06:28:58.656331 (+ 63us) client_negotiation.cc:240] Negotiation successful
0806 06:28:58.656383 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":897,"thread_start_us":93,"threads_started":1}
I20260806 06:28:58.656837 10447 tablet_service.cc:1514] Processing DeleteTablet for tablet 3a80c47c38454f9a912330f3f42c76ba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 06:28:58 UTC) from {username='kudu', principal='kudu/127.21.12.190@KRBTEST.COM'} at 127.0.0.1:42276
I20260806 06:28:58.657402 10558 tablet_replica.cc:330] stopping tablet replica
I20260806 06:28:58.657488 10558 raft_consensus.cc:2229] T 3a80c47c38454f9a912330f3f42c76ba P e7a2421395c9473e91a329c11588a784 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 06:28:58.657554 10558 raft_consensus.cc:2258] T 3a80c47c38454f9a912330f3f42c76ba P e7a2421395c9473e91a329c11588a784 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 06:28:58.657827 10558 ts_tablet_manager.cc:1905] T 3a80c47c38454f9a912330f3f42c76ba P e7a2421395c9473e91a329c11588a784: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 06:28:58.659859 10558 ts_tablet_manager.cc:1918] T 3a80c47c38454f9a912330f3f42c76ba P e7a2421395c9473e91a329c11588a784: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 06:28:58.659921 10558 log.cc:1196] T 3a80c47c38454f9a912330f3f42c76ba P e7a2421395c9473e91a329c11588a784: Deleting WAL directory at /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/minicluster-data/ts-0/wal/wals/3a80c47c38454f9a912330f3f42c76ba
I20260806 06:28:58.660111 10558 ts_tablet_manager.cc:1939] T 3a80c47c38454f9a912330f3f42c76ba P e7a2421395c9473e91a329c11588a784: Deleting consensus metadata
I20260806 06:28:58.660543 10231 catalog_manager.cc:4841] TS e7a2421395c9473e91a329c11588a784 (127.21.12.129:42271): tablet 3a80c47c38454f9a912330f3f42c76ba (table test-table [id=5d36c565a0df43a0a73bfb9292c6ac78]) successfully deleted
W20260806 06:28:58.727010 10552 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 06:28:58.727224 10552 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 06:28:58.727254 10552 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 06:28:58.728384 10552 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 06:28:58.728427 10552 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 06:28:58.728462 10552 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.12.130
I20260806 06:28:58.729890 10552 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.148:41671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.130.keytab
--principal=kudu/127.21.12.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.12.130:43341
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.12.130
--webserver_port=43723
--tserver_master_addrs=127.21.12.190:36809
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10552
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.12.130
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 05:11:02 UTC on ccycloud.quasar-hvzbbz.root.comops.site
build id 4569617
I20260806 06:28:58.730124 10552 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 06:28:58.730371 10552 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 06:28:58.735009 10571 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:28:58.735028 10566 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:28:58.735010 10567 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:28:58.735416 10552 server_base.cc:1034] running on GCE node
I20260806 06:28:58.735646 10552 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 06:28:58.735868 10552 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 06:28:58.737003 10552 hybrid_clock.cc:648] HybridClock initialized: now 1785997738736991 us; error 28 us; skew 500 ppm
Aug 06 06:28:58 dist-test-slave-4tdj krb5kdc[9417](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997738, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 06:28:58.739370 10552 init.cc:377] Logged in from keytab as kudu/127.21.12.130@KRBTEST.COM (short username kudu)
I20260806 06:28:58.739753 10552 webserver.cc:460] Webserver started at http://127.21.12.130:43723/ using document root <none> and password file <none>
I20260806 06:28:58.739925 10552 fs_manager.cc:359] Metadata directory not provided
I20260806 06:28:58.739975 10552 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 06:28:58.741272 10552 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260806 06:28:58.741878 10580 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 06:28:58.742022 10552 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260806 06:28:58.742115 10552 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/minicluster-data/ts-1/wal
uuid: "a397a32a5c4c4681951c1de266125f9c"
format_stamp: "Formatted at 2026-08-06 06:28:58 on dist-test-slave-4tdj"
server_key: "de85f746c85d5607dfe6f4a622f7693c"
server_key_iv: "0c0d6c91ab3445aad3773bb0edae0806"
server_key_version: "encryptionkey@0"
I20260806 06:28:58.742470 10552 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 06:28:58.752926 10552 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 06:28:58.754797 10552 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 06:28:58.754920 10552 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 06:28:58.755112 10552 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 06:28:58.755383 10552 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 06:28:58.755415 10552 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 06:28:58.755443 10552 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 06:28:58.755467 10552 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 06:28:58.760757 10552 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.130:43341
I20260806 06:28:58.760834 10764 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.130:43341 every 8 connection(s)
I20260806 06:28:58.761116 10552 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/minicluster-data/ts-1/data/info.pb
Aug 06 06:28:58 dist-test-slave-4tdj krb5kdc[9417](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997738, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.130@KRBTEST.COM for kudu/127.21.12.190@KRBTEST.COM
I20260806 06:28:58.768978 21554 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHhLhT/build/release/bin/kudu as pid 10552
I20260806 06:28:58.769279 21554 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHhLhT/build/release/bin/kudu
/tmp/dist-test-taskFHhLhT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.12.131:32863
--local_ip_for_outbound_sockets=127.21.12.131
--tserver_master_addrs=127.21.12.190:36809
--webserver_port=42145
--webserver_interface=127.21.12.131
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.131.keytab
--principal=kudu/127.21.12.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.148:41671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 06:28:58.769445 10549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:28:58.761810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.130:47163 (local address 127.21.12.190:36809)
0806 06:28:58.761886 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0806 06:28:58.761888 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:28:58.762012 (+ 124us) server_negotiation.cc:407] Connection header received
0806 06:28:58.762512 (+ 500us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:28:58.762513 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:28:58.762527 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:28:58.762572 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:28:58.763449 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:28:58.763934 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:28:58.764847 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:28:58.765038 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:28:58.765087 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:28:58.767047 (+ 1960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:28:58.767049 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:28:58.767049 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 06:28:58.767056 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:28:58.768217 (+ 1161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:28:58.768572 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:28:58.768572 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:28:58.768573 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:28:58.768621 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:28:58.768792 (+ 171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:28:58.768793 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:28:58.768793 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:28:58.768893 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:28:58.768936 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0806 06:28:58.769260 (+ 324us) server_negotiation.cc:299] Negotiation successful
0806 06:28:58.769342 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260806 06:28:58.769579 10767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:28:58.761768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:36809 (local address 127.21.12.130:47163)
0806 06:28:58.761901 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0806 06:28:58.761935 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0806 06:28:58.762420 (+ 485us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:28:58.762634 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:28:58.762638 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:28:58.763190 (+ 552us) client_negotiation.cc:189] Negotiated authn=SASL
0806 06:28:58.763371 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:28:58.763376 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:28:58.764012 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:28:58.764013 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:28:58.764786 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:28:58.764788 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:28:58.765096 (+ 308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:28:58.765097 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:28:58.765185 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:28:58.765643 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 06:28:58.765652 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0806 06:28:58.766963 (+ 1311us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 06:28:58.768305 (+ 1342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:28:58.768311 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:28:58.768313 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:28:58.768512 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:28:58.768668 (+ 156us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:28:58.768668 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:28:58.768669 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:28:58.768752 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:28:58.768945 (+ 193us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 06:28:58.768950 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 06:28:58.769102 (+ 152us) client_negotiation.cc:769] Sending connection context
0806 06:28:58.769233 (+ 131us) client_negotiation.cc:240] Negotiation successful
0806 06:28:58.769387 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":65,"threads_started":1}
I20260806 06:28:58.771211 10765 heartbeater.cc:347] Connected to a master server at 127.21.12.190:36809
I20260806 06:28:58.771313 10765 heartbeater.cc:464] Registering TS with master...
I20260806 06:28:58.771490 10765 heartbeater.cc:511] Master 127.21.12.190:36809 requested a full tablet report, sending...
I20260806 06:28:58.771862 10262 ts_manager.cc:194] Registered new tserver with Master: a397a32a5c4c4681951c1de266125f9c (127.21.12.130:43341)
I20260806 06:28:58.772286 10262 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.12.130@KRBTEST.COM'} at 127.21.12.130:47163
W20260806 06:28:58.850955 10768 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 06:28:58.851145 10768 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 06:28:58.851166 10768 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 06:28:58.852260 10768 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 06:28:58.852301 10768 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 06:28:58.852325 10768 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.12.131
I20260806 06:28:58.853730 10768 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.148:41671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.131.keytab
--principal=kudu/127.21.12.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.12.131:32863
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.12.131
--webserver_port=42145
--tserver_master_addrs=127.21.12.190:36809
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10768
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.12.131
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 05:11:02 UTC on ccycloud.quasar-hvzbbz.root.comops.site
build id 4569617
I20260806 06:28:58.853942 10768 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 06:28:58.854132 10768 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 06:28:58.859010 10775 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:28:58.859019 10779 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:28:58.859035 10776 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:28:58.859035 10768 server_base.cc:1034] running on GCE node
I20260806 06:28:58.859457 10768 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 06:28:58.859700 10768 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 06:28:58.860837 10768 hybrid_clock.cc:648] HybridClock initialized: now 1785997738860827 us; error 31 us; skew 500 ppm
Aug 06 06:28:58 dist-test-slave-4tdj krb5kdc[9417](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997738, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 06:28:58.863306 10768 init.cc:377] Logged in from keytab as kudu/127.21.12.131@KRBTEST.COM (short username kudu)
I20260806 06:28:58.863695 10768 webserver.cc:460] Webserver started at http://127.21.12.131:42145/ using document root <none> and password file <none>
I20260806 06:28:58.863881 10768 fs_manager.cc:359] Metadata directory not provided
I20260806 06:28:58.863942 10768 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 06:28:58.865221 10768 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260806 06:28:58.865770 10793 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 06:28:58.865928 10768 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260806 06:28:58.866012 10768 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/minicluster-data/ts-2/wal
uuid: "b778fbcc33a740b0876dd34c9fa6829c"
format_stamp: "Formatted at 2026-08-06 06:28:58 on dist-test-slave-4tdj"
server_key: "c10e6fb753d0a8555df8b25ce0e3e3da"
server_key_iv: "a2995c2452fb6ac189c123a98f246f68"
server_key_version: "encryptionkey@0"
I20260806 06:28:58.866343 10768 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 06:28:58.881412 10768 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 06:28:58.883443 10768 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 06:28:58.883580 10768 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 06:28:58.883841 10768 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 06:28:58.884177 10768 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 06:28:58.884223 10768 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 06:28:58.884296 10768 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 06:28:58.884317 10768 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 06:28:58.890350 10768 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.131:32863
I20260806 06:28:58.890407 10957 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.131:32863 every 8 connection(s)
I20260806 06:28:58.890714 10768 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/minicluster-data/ts-2/data/info.pb
I20260806 06:28:58.894892 21554 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHhLhT/build/release/bin/kudu as pid 10768
Aug 06 06:28:58 dist-test-slave-4tdj krb5kdc[9417](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997738, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.131@KRBTEST.COM for kudu/127.21.12.190@KRBTEST.COM
I20260806 06:28:58.899506 10549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:28:58.891373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.131:60215 (local address 127.21.12.190:36809)
0806 06:28:58.891429 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0806 06:28:58.891431 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:28:58.891615 (+ 184us) server_negotiation.cc:407] Connection header received
0806 06:28:58.892071 (+ 456us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:28:58.892072 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:28:58.892082 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:28:58.892138 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:28:58.892975 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:28:58.893475 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:28:58.894413 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:28:58.894601 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:28:58.894653 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:28:58.896764 (+ 2111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:28:58.896765 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:28:58.896766 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 06:28:58.896773 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:28:58.898066 (+ 1293us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:28:58.898500 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:28:58.898502 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:28:58.898502 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:28:58.898558 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:28:58.898826 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:28:58.898827 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:28:58.898828 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:28:58.898950 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:28:58.899019 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0806 06:28:58.899330 (+ 311us) server_negotiation.cc:299] Negotiation successful
0806 06:28:58.899410 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260806 06:28:58.899679 10960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:28:58.891361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:36809 (local address 127.21.12.131:60215)
0806 06:28:58.891503 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0806 06:28:58.891543 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0806 06:28:58.891994 (+ 451us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:28:58.892190 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:28:58.892194 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:28:58.892727 (+ 533us) client_negotiation.cc:189] Negotiated authn=SASL
0806 06:28:58.892908 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:28:58.892912 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:28:58.893535 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:28:58.893536 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:28:58.894351 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:28:58.894353 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:28:58.894658 (+ 305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:28:58.894659 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:28:58.894759 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:28:58.895195 (+ 436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 06:28:58.895206 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0806 06:28:58.896681 (+ 1475us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 06:28:58.898138 (+ 1457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:28:58.898145 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:28:58.898148 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:28:58.898437 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:28:58.898647 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:28:58.898648 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:28:58.898649 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:28:58.898762 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:28:58.899026 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 06:28:58.899034 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 06:28:58.899188 (+ 154us) client_negotiation.cc:769] Sending connection context
0806 06:28:58.899308 (+ 120us) client_negotiation.cc:240] Negotiation successful
0806 06:28:58.899445 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":62,"threads_started":1}
I20260806 06:28:58.900130 10958 heartbeater.cc:347] Connected to a master server at 127.21.12.190:36809
I20260806 06:28:58.900213 10958 heartbeater.cc:464] Registering TS with master...
I20260806 06:28:58.900379 10958 heartbeater.cc:511] Master 127.21.12.190:36809 requested a full tablet report, sending...
I20260806 06:28:58.900772 10262 ts_manager.cc:194] Registered new tserver with Master: b778fbcc33a740b0876dd34c9fa6829c (127.21.12.131:32863)
I20260806 06:28:58.901211 10262 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.12.131@KRBTEST.COM'} at 127.21.12.131:60215
I20260806 06:28:58.901607 10962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:28:58.895239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39556 (local address 127.21.12.190:36809)
0806 06:28:58.895343 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0806 06:28:58.895345 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:28:58.895352 (+ 7us) server_negotiation.cc:407] Connection header received
0806 06:28:58.895415 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:28:58.895416 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:28:58.895427 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:28:58.895481 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:28:58.896002 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:28:58.896631 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:28:58.897597 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:28:58.897785 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:28:58.897834 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:28:58.899060 (+ 1226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:28:58.899061 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:28:58.899062 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 06:28:58.899069 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:28:58.900608 (+ 1539us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:28:58.900927 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:28:58.900928 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:28:58.900928 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:28:58.900988 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:28:58.901139 (+ 151us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:28:58.901139 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:28:58.901140 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:28:58.901244 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:28:58.901401 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0806 06:28:58.901438 (+ 37us) server_negotiation.cc:299] Negotiation successful
0806 06:28:58.901517 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84,"thread_start_us":57,"threads_started":1}
I20260806 06:28:58.902565 21554 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 06:28:58.904877 10962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:28:58.902869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39558 (local address 127.21.12.190:36809)
0806 06:28:58.902912 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0806 06:28:58.902914 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:28:58.902981 (+ 67us) server_negotiation.cc:407] Connection header received
0806 06:28:58.903040 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:28:58.903041 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:28:58.903067 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:28:58.903124 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 06:28:58.903200 (+ 76us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:28:58.903612 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:28:58.904255 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:28:58.904426 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:28:58.904465 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:28:58.904662 (+ 197us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 06:28:58.904744 (+ 82us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 06:28:58.904793 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0806 06:28:58.904820 (+ 27us) server_negotiation.cc:299] Negotiation successful
0806 06:28:58.904835 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260806 06:28:58.905195 10262 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:39558:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 06:28:58.906872 10262 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 06:28:58.908946 10447 tablet_service.cc:1467] Processing CreateTablet for tablet cc231401bc374df3a18d5e039b36a2e9 (DEFAULT_TABLE table=test-table [id=6821954238e54e73b36e296119fa1fb4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 06:28:58.909139 10447 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc231401bc374df3a18d5e039b36a2e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 06:28:58.910974 10384 tablet_bootstrap.cc:492] T cc231401bc374df3a18d5e039b36a2e9 P e7a2421395c9473e91a329c11588a784: Bootstrap starting.
I20260806 06:28:58.911332 10384 tablet_bootstrap.cc:654] T cc231401bc374df3a18d5e039b36a2e9 P e7a2421395c9473e91a329c11588a784: Neither blocks nor log segments found. Creating new log.
I20260806 06:28:58.912029 10384 tablet_bootstrap.cc:492] T cc231401bc374df3a18d5e039b36a2e9 P e7a2421395c9473e91a329c11588a784: No bootstrap required, opened a new log
I20260806 06:28:58.912081 10384 ts_tablet_manager.cc:1397] T cc231401bc374df3a18d5e039b36a2e9 P e7a2421395c9473e91a329c11588a784: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260806 06:28:58.912143 10554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:28:58.908471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.131:32863 (local address 127.0.0.1:48960)
0806 06:28:58.908513 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0806 06:28:58.908522 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0806 06:28:58.908556 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:28:58.908951 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:28:58.908952 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:28:58.908953 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 06:28:58.908988 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:28:58.908990 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:28:58.909988 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:28:58.909990 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:28:58.911047 (+ 1057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:28:58.911049 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:28:58.911949 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:28:58.911950 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:28:58.911996 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:28:58.911998 (+ 2us) client_negotiation.cc:769] Sending connection context
0806 06:28:58.912041 (+ 43us) client_negotiation.cc:240] Negotiation successful
0806 06:28:58.912082 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260806 06:28:58.912186 10965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:28:58.908562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48960 (local address 127.21.12.131:32863)
0806 06:28:58.908807 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0806 06:28:58.908812 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 06:28:58.908822 (+ 10us) server_negotiation.cc:407] Connection header received
0806 06:28:58.908855 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:28:58.908857 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:28:58.908871 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:28:58.908940 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 06:28:58.909065 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:28:58.909882 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:28:58.911131 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:28:58.911798 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:28:58.911861 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:28:58.911918 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0806 06:28:58.912069 (+ 151us) server_negotiation.cc:299] Negotiation successful
0806 06:28:58.912107 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":211,"thread_start_us":86,"threads_started":1}
I20260806 06:28:58.912194 10384 raft_consensus.cc:348] T cc231401bc374df3a18d5e039b36a2e9 P e7a2421395c9473e91a329c11588a784 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b778fbcc33a740b0876dd34c9fa6829c" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 32863 } } peers { permanent_uuid: "a397a32a5c4c4681951c1de266125f9c" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 43341 } } peers { permanent_uuid: "e7a2421395c9473e91a329c11588a784" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 42271 } }
I20260806 06:28:58.912266 10384 raft_consensus.cc:374] T cc231401bc374df3a18d5e039b36a2e9 P e7a2421395c9473e91a329c11588a784 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 06:28:58.912317 10384 raft_consensus.cc:729] T cc231401bc374df3a18d5e039b36a2e9 P e7a2421395c9473e91a329c11588a784 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7a2421395c9473e91a329c11588a784, State: Initialized, Role: FOLLOWER
I20260806 06:28:58.912375 10384 consensus_queue.cc:260] T cc231401bc374df3a18d5e039b36a2e9 P e7a2421395c9473e91a329c11588a784 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b778fbcc33a740b0876dd34c9fa6829c" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 32863 } } peers { permanent_uuid: "a397a32a5c4c4681951c1de266125f9c" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 43341 } } peers { permanent_uuid: "e7a2421395c9473e91a329c11588a784" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 42271 } }
I20260806 06:28:58.912489 10384 ts_tablet_manager.cc:1428] T cc231401bc374df3a18d5e039b36a2e9 P e7a2421395c9473e91a329c11588a784: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260806 06:28:58.912806 10863 tablet_service.cc:1467] Processing CreateTablet for tablet cc231401bc374df3a18d5e039b36a2e9 (DEFAULT_TABLE table=test-table [id=6821954238e54e73b36e296119fa1fb4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 06:28:58.913157 10863 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc231401bc374df3a18d5e039b36a2e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 06:28:58.914455 10966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:28:58.908557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.130:43341 (local address 127.0.0.1:51804)
0806 06:28:58.908708 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0806 06:28:58.908718 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0806 06:28:58.908771 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:28:58.911030 (+ 2259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:28:58.911031 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:28:58.911034 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 06:28:58.911071 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:28:58.911072 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:28:58.912256 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:28:58.912257 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:28:58.913323 (+ 1066us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:28:58.913326 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:28:58.914193 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:28:58.914194 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:28:58.914259 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:28:58.914263 (+ 4us) client_negotiation.cc:769] Sending connection context
0806 06:28:58.914348 (+ 85us) client_negotiation.cc:240] Negotiation successful
0806 06:28:58.914392 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":105,"threads_started":1}
I20260806 06:28:58.914495 10967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:28:58.908664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51804 (local address 127.21.12.130:43341)
0806 06:28:58.909060 (+ 396us) server_negotiation.cc:206] Beginning negotiation
0806 06:28:58.909064 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 06:28:58.909073 (+ 9us) server_negotiation.cc:407] Connection header received
0806 06:28:58.909108 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:28:58.909110 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:28:58.909126 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:28:58.909185 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 06:28:58.911162 (+ 1977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:28:58.912128 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:28:58.913406 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:28:58.914080 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:28:58.914154 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:28:58.914207 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0806 06:28:58.914372 (+ 165us) server_negotiation.cc:299] Negotiation successful
0806 06:28:58.914418 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":98,"threads_started":1}
I20260806 06:28:58.915117 10652 tablet_service.cc:1467] Processing CreateTablet for tablet cc231401bc374df3a18d5e039b36a2e9 (DEFAULT_TABLE table=test-table [id=6821954238e54e73b36e296119fa1fb4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 06:28:58.915442 10652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc231401bc374df3a18d5e039b36a2e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 06:28:58.916253 10969 tablet_bootstrap.cc:492] T cc231401bc374df3a18d5e039b36a2e9 P b778fbcc33a740b0876dd34c9fa6829c: Bootstrap starting.
I20260806 06:28:58.916883 10969 tablet_bootstrap.cc:654] T cc231401bc374df3a18d5e039b36a2e9 P b778fbcc33a740b0876dd34c9fa6829c: Neither blocks nor log segments found. Creating new log.
I20260806 06:28:58.917321 10969 log.cc:824] T cc231401bc374df3a18d5e039b36a2e9 P b778fbcc33a740b0876dd34c9fa6829c: Log is configured to *not* fsync() on all Append() calls
I20260806 06:28:58.918025 10969 tablet_bootstrap.cc:492] T cc231401bc374df3a18d5e039b36a2e9 P b778fbcc33a740b0876dd34c9fa6829c: No bootstrap required, opened a new log
I20260806 06:28:58.918085 10969 ts_tablet_manager.cc:1397] T cc231401bc374df3a18d5e039b36a2e9 P b778fbcc33a740b0876dd34c9fa6829c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260806 06:28:58.918821 10971 tablet_bootstrap.cc:492] T cc231401bc374df3a18d5e039b36a2e9 P a397a32a5c4c4681951c1de266125f9c: Bootstrap starting.
I20260806 06:28:58.919373 10971 tablet_bootstrap.cc:654] T cc231401bc374df3a18d5e039b36a2e9 P a397a32a5c4c4681951c1de266125f9c: Neither blocks nor log segments found. Creating new log.
I20260806 06:28:58.919600 10969 raft_consensus.cc:348] T cc231401bc374df3a18d5e039b36a2e9 P b778fbcc33a740b0876dd34c9fa6829c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b778fbcc33a740b0876dd34c9fa6829c" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 32863 } } peers { permanent_uuid: "a397a32a5c4c4681951c1de266125f9c" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 43341 } } peers { permanent_uuid: "e7a2421395c9473e91a329c11588a784" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 42271 } }
I20260806 06:28:58.919732 10969 raft_consensus.cc:374] T cc231401bc374df3a18d5e039b36a2e9 P b778fbcc33a740b0876dd34c9fa6829c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 06:28:58.919766 10969 raft_consensus.cc:729] T cc231401bc374df3a18d5e039b36a2e9 P b778fbcc33a740b0876dd34c9fa6829c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b778fbcc33a740b0876dd34c9fa6829c, State: Initialized, Role: FOLLOWER
I20260806 06:28:58.919826 10971 log.cc:824] T cc231401bc374df3a18d5e039b36a2e9 P a397a32a5c4c4681951c1de266125f9c: Log is configured to *not* fsync() on all Append() calls
I20260806 06:28:58.919878 10969 consensus_queue.cc:260] T cc231401bc374df3a18d5e039b36a2e9 P b778fbcc33a740b0876dd34c9fa6829c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b778fbcc33a740b0876dd34c9fa6829c" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 32863 } } peers { permanent_uuid: "a397a32a5c4c4681951c1de266125f9c" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 43341 } } peers { permanent_uuid: "e7a2421395c9473e91a329c11588a784" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 42271 } }
I20260806 06:28:58.920158 10969 ts_tablet_manager.cc:1428] T cc231401bc374df3a18d5e039b36a2e9 P b778fbcc33a740b0876dd34c9fa6829c: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260806 06:28:58.920167 10958 heartbeater.cc:503] Master 127.21.12.190:36809 was elected leader, sending a full tablet report...
I20260806 06:28:58.920667 10971 tablet_bootstrap.cc:492] T cc231401bc374df3a18d5e039b36a2e9 P a397a32a5c4c4681951c1de266125f9c: No bootstrap required, opened a new log
I20260806 06:28:58.920751 10971 ts_tablet_manager.cc:1397] T cc231401bc374df3a18d5e039b36a2e9 P a397a32a5c4c4681951c1de266125f9c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260806 06:28:58.922179 10971 raft_consensus.cc:348] T cc231401bc374df3a18d5e039b36a2e9 P a397a32a5c4c4681951c1de266125f9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b778fbcc33a740b0876dd34c9fa6829c" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 32863 } } peers { permanent_uuid: "a397a32a5c4c4681951c1de266125f9c" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 43341 } } peers { permanent_uuid: "e7a2421395c9473e91a329c11588a784" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 42271 } }
I20260806 06:28:58.922353 10971 raft_consensus.cc:374] T cc231401bc374df3a18d5e039b36a2e9 P a397a32a5c4c4681951c1de266125f9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 06:28:58.922399 10971 raft_consensus.cc:729] T cc231401bc374df3a18d5e039b36a2e9 P a397a32a5c4c4681951c1de266125f9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a397a32a5c4c4681951c1de266125f9c, State: Initialized, Role: FOLLOWER
I20260806 06:28:58.922508 10971 consensus_queue.cc:260] T cc231401bc374df3a18d5e039b36a2e9 P a397a32a5c4c4681951c1de266125f9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b778fbcc33a740b0876dd34c9fa6829c" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 32863 } } peers { permanent_uuid: "a397a32a5c4c4681951c1de266125f9c" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 43341 } } peers { permanent_uuid: "e7a2421395c9473e91a329c11588a784" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 42271 } }
I20260806 06:28:58.922739 10971 ts_tablet_manager.cc:1428] T cc231401bc374df3a18d5e039b36a2e9 P a397a32a5c4c4681951c1de266125f9c: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260806 06:28:58.922773 10765 heartbeater.cc:503] Master 127.21.12.190:36809 was elected leader, sending a full tablet report...
I20260806 06:28:58.938779 10978 raft_consensus.cc:482] T cc231401bc374df3a18d5e039b36a2e9 P a397a32a5c4c4681951c1de266125f9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 06:28:58.938866 10978 raft_consensus.cc:504] T cc231401bc374df3a18d5e039b36a2e9 P a397a32a5c4c4681951c1de266125f9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b778fbcc33a740b0876dd34c9fa6829c" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 32863 } } peers { permanent_uuid: "a397a32a5c4c4681951c1de266125f9c" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 43341 } } peers { permanent_uuid: "e7a2421395c9473e91a329c11588a784" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 42271 } }
I20260806 06:28:58.939152 10978 leader_election.cc:288] T cc231401bc374df3a18d5e039b36a2e9 P a397a32a5c4c4681951c1de266125f9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b778fbcc33a740b0876dd34c9fa6829c (127.21.12.131:32863), e7a2421395c9473e91a329c11588a784 (127.21.12.129:42271)
I20260806 06:28:58.941845 10965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:28:58.939372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.130:34349 (local address 127.21.12.131:32863)
0806 06:28:58.939413 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0806 06:28:58.939415 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:28:58.939446 (+ 31us) server_negotiation.cc:407] Connection header received
0806 06:28:58.939466 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:28:58.939466 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:28:58.939474 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:28:58.939525 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 06:28:58.939710 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:28:58.940261 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:28:58.941165 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:28:58.941539 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:28:58.941598 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:28:58.941606 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0806 06:28:58.941754 (+ 148us) server_negotiation.cc:299] Negotiation successful
0806 06:28:58.941787 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260806 06:28:58.941845 10555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:28:58.939568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.130:33061 (local address 127.21.12.129:42271)
0806 06:28:58.939623 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0806 06:28:58.939625 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:28:58.939633 (+ 8us) server_negotiation.cc:407] Connection header received
0806 06:28:58.939657 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:28:58.939658 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:28:58.939669 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:28:58.939720 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 06:28:58.939856 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:28:58.940331 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:28:58.941195 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:28:58.941539 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:28:58.941599 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:28:58.941608 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0806 06:28:58.941739 (+ 131us) server_negotiation.cc:299] Negotiation successful
0806 06:28:58.941776 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260806 06:28:58.941861 10767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:28:58.939314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.131:32863 (local address 127.21.12.130:34349)
0806 06:28:58.939357 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0806 06:28:58.939367 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0806 06:28:58.939404 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:28:58.939555 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:28:58.939557 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:28:58.939559 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 06:28:58.939635 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:28:58.939637 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:28:58.940335 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:28:58.940335 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:28:58.941085 (+ 750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:28:58.941088 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:28:58.941614 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:28:58.941615 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:28:58.941692 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:28:58.941694 (+ 2us) client_negotiation.cc:769] Sending connection context
0806 06:28:58.941742 (+ 48us) client_negotiation.cc:240] Negotiation successful
0806 06:28:58.941779 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24,"mutex_wait_us":1}
I20260806 06:28:58.941869 10980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:28:58.939417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.129:42271 (local address 127.21.12.130:33061)
0806 06:28:58.939555 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0806 06:28:58.939564 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0806 06:28:58.939605 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:28:58.939728 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:28:58.939729 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:28:58.939730 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 06:28:58.939798 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:28:58.939800 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:28:58.940394 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:28:58.940394 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:28:58.941112 (+ 718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:28:58.941114 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:28:58.941607 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:28:58.941608 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:28:58.941682 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:28:58.941685 (+ 3us) client_negotiation.cc:769] Sending connection context
0806 06:28:58.941725 (+ 40us) client_negotiation.cc:240] Negotiation successful
0806 06:28:58.941762 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"mutex_wait_us":2,"thread_start_us":84,"threads_started":1}
I20260806 06:28:58.942193 10472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc231401bc374df3a18d5e039b36a2e9" candidate_uuid: "a397a32a5c4c4681951c1de266125f9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7a2421395c9473e91a329c11588a784" is_pre_election: true
I20260806 06:28:58.942210 10893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc231401bc374df3a18d5e039b36a2e9" candidate_uuid: "a397a32a5c4c4681951c1de266125f9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b778fbcc33a740b0876dd34c9fa6829c" is_pre_election: true
I20260806 06:28:58.942353 10472 raft_consensus.cc:2454] T cc231401bc374df3a18d5e039b36a2e9 P e7a2421395c9473e91a329c11588a784 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a397a32a5c4c4681951c1de266125f9c in term 0.
I20260806 06:28:58.942358 10893 raft_consensus.cc:2454] T cc231401bc374df3a18d5e039b36a2e9 P b778fbcc33a740b0876dd34c9fa6829c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a397a32a5c4c4681951c1de266125f9c in term 0.
I20260806 06:28:58.942548 10587 leader_election.cc:302] T cc231401bc374df3a18d5e039b36a2e9 P a397a32a5c4c4681951c1de266125f9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a397a32a5c4c4681951c1de266125f9c, e7a2421395c9473e91a329c11588a784; no voters:
I20260806 06:28:58.942654 10978 raft_consensus.cc:2781] T cc231401bc374df3a18d5e039b36a2e9 P a397a32a5c4c4681951c1de266125f9c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 06:28:58.942708 10978 raft_consensus.cc:482] T cc231401bc374df3a18d5e039b36a2e9 P a397a32a5c4c4681951c1de266125f9c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 06:28:58.942734 10978 raft_consensus.cc:3037] T cc231401bc374df3a18d5e039b36a2e9 P a397a32a5c4c4681951c1de266125f9c [term 0 FOLLOWER]: Advancing to term 1
I20260806 06:28:58.943331 10978 raft_consensus.cc:504] T cc231401bc374df3a18d5e039b36a2e9 P a397a32a5c4c4681951c1de266125f9c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b778fbcc33a740b0876dd34c9fa6829c" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 32863 } } peers { permanent_uuid: "a397a32a5c4c4681951c1de266125f9c" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 43341 } } peers { permanent_uuid: "e7a2421395c9473e91a329c11588a784" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 42271 } }
I20260806 06:28:58.943424 10978 leader_election.cc:288] T cc231401bc374df3a18d5e039b36a2e9 P a397a32a5c4c4681951c1de266125f9c [CANDIDATE]: Term 1 election: Requested vote from peers b778fbcc33a740b0876dd34c9fa6829c (127.21.12.131:32863), e7a2421395c9473e91a329c11588a784 (127.21.12.129:42271)
I20260806 06:28:58.943586 10893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc231401bc374df3a18d5e039b36a2e9" candidate_uuid: "a397a32a5c4c4681951c1de266125f9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b778fbcc33a740b0876dd34c9fa6829c"
I20260806 06:28:58.943615 10472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc231401bc374df3a18d5e039b36a2e9" candidate_uuid: "a397a32a5c4c4681951c1de266125f9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7a2421395c9473e91a329c11588a784"
I20260806 06:28:58.943663 10893 raft_consensus.cc:3037] T cc231401bc374df3a18d5e039b36a2e9 P b778fbcc33a740b0876dd34c9fa6829c [term 0 FOLLOWER]: Advancing to term 1
I20260806 06:28:58.943676 10472 raft_consensus.cc:3037] T cc231401bc374df3a18d5e039b36a2e9 P e7a2421395c9473e91a329c11588a784 [term 0 FOLLOWER]: Advancing to term 1
I20260806 06:28:58.944492 10893 raft_consensus.cc:2454] T cc231401bc374df3a18d5e039b36a2e9 P b778fbcc33a740b0876dd34c9fa6829c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a397a32a5c4c4681951c1de266125f9c in term 1.
I20260806 06:28:58.944691 10587 leader_election.cc:302] T cc231401bc374df3a18d5e039b36a2e9 P a397a32a5c4c4681951c1de266125f9c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a397a32a5c4c4681951c1de266125f9c, b778fbcc33a740b0876dd34c9fa6829c; no voters:
I20260806 06:28:58.944773 10978 raft_consensus.cc:2781] T cc231401bc374df3a18d5e039b36a2e9 P a397a32a5c4c4681951c1de266125f9c [term 1 FOLLOWER]: Leader election won for term 1
I20260806 06:28:58.944837 10472 raft_consensus.cc:2454] T cc231401bc374df3a18d5e039b36a2e9 P e7a2421395c9473e91a329c11588a784 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a397a32a5c4c4681951c1de266125f9c in term 1.
I20260806 06:28:58.944904 10978 raft_consensus.cc:686] T cc231401bc374df3a18d5e039b36a2e9 P a397a32a5c4c4681951c1de266125f9c [term 1 LEADER]: Becoming Leader. State: Replica: a397a32a5c4c4681951c1de266125f9c, State: Running, Role: LEADER
I20260806 06:28:58.944993 10978 consensus_queue.cc:237] T cc231401bc374df3a18d5e039b36a2e9 P a397a32a5c4c4681951c1de266125f9c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b778fbcc33a740b0876dd34c9fa6829c" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 32863 } } peers { permanent_uuid: "a397a32a5c4c4681951c1de266125f9c" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 43341 } } peers { permanent_uuid: "e7a2421395c9473e91a329c11588a784" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 42271 } }
I20260806 06:28:58.945724 10262 catalog_manager.cc:5495] T cc231401bc374df3a18d5e039b36a2e9 P a397a32a5c4c4681951c1de266125f9c reported cstate change: term changed from 0 to 1, leader changed from <none> to a397a32a5c4c4681951c1de266125f9c (127.21.12.130). New cstate: current_term: 1 leader_uuid: "a397a32a5c4c4681951c1de266125f9c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b778fbcc33a740b0876dd34c9fa6829c" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 32863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a397a32a5c4c4681951c1de266125f9c" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 43341 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e7a2421395c9473e91a329c11588a784" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 42271 } health_report { overall_health: UNKNOWN } } }
I20260806 06:28:58.959260 10967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:28:58.957135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51806 (local address 127.21.12.130:43341)
0806 06:28:58.957207 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0806 06:28:58.957209 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:28:58.957217 (+ 8us) server_negotiation.cc:407] Connection header received
0806 06:28:58.957247 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:28:58.957248 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:28:58.957265 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:28:58.957324 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 06:28:58.957424 (+ 100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:28:58.957868 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:28:58.958574 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:28:58.958772 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:28:58.958824 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:28:58.958950 (+ 126us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 06:28:58.959043 (+ 93us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 06:28:58.959108 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0806 06:28:58.959145 (+ 37us) server_negotiation.cc:299] Negotiation successful
0806 06:28:58.959163 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260806 06:28:58.960839 10893 raft_consensus.cc:1261] T cc231401bc374df3a18d5e039b36a2e9 P b778fbcc33a740b0876dd34c9fa6829c [term 1 FOLLOWER]: Refusing update from remote peer a397a32a5c4c4681951c1de266125f9c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 06:28:58.960840 10472 raft_consensus.cc:1261] T cc231401bc374df3a18d5e039b36a2e9 P e7a2421395c9473e91a329c11588a784 [term 1 FOLLOWER]: Refusing update from remote peer a397a32a5c4c4681951c1de266125f9c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 06:28:58.961163 10978 consensus_queue.cc:1035] T cc231401bc374df3a18d5e039b36a2e9 P a397a32a5c4c4681951c1de266125f9c [LEADER]: Connected to new peer: Peer: permanent_uuid: "b778fbcc33a740b0876dd34c9fa6829c" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 32863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 06:28:58.961301 10981 consensus_queue.cc:1035] T cc231401bc374df3a18d5e039b36a2e9 P a397a32a5c4c4681951c1de266125f9c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7a2421395c9473e91a329c11588a784" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 42271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 06:28:58.963330 10989 mvcc.cc:205] Tried to move back new op lower bound from 7315446738781245440 to 7315446738719309824. Current Snapshot: MvccSnapshot[applied={T|T < 7315446738781245440}]
I20260806 06:28:58.964006 10991 mvcc.cc:205] Tried to move back new op lower bound from 7315446738781245440 to 7315446738719309824. Current Snapshot: MvccSnapshot[applied={T|T < 7315446738781245440}]
I20260806 06:28:58.965412 10261 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:39558:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 06:28:58.965497 10261 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:39558:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 06:28:58.966179 10261 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2ad50aad27464833af0c85627af4d871: Sending DeleteTablet for 3 replicas of tablet cc231401bc374df3a18d5e039b36a2e9
I20260806 06:28:58.966526 10652 tablet_service.cc:1514] Processing DeleteTablet for tablet cc231401bc374df3a18d5e039b36a2e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 06:28:58 UTC) from {username='kudu', principal='kudu/127.21.12.190@KRBTEST.COM'} at 127.0.0.1:51804
I20260806 06:28:58.966526 10863 tablet_service.cc:1514] Processing DeleteTablet for tablet cc231401bc374df3a18d5e039b36a2e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 06:28:58 UTC) from {username='kudu', principal='kudu/127.21.12.190@KRBTEST.COM'} at 127.0.0.1:48960
I20260806 06:28:58.966710 10447 tablet_service.cc:1514] Processing DeleteTablet for tablet cc231401bc374df3a18d5e039b36a2e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 06:28:58 UTC) from {username='kudu', principal='kudu/127.21.12.190@KRBTEST.COM'} at 127.0.0.1:42276
I20260806 06:28:58.966763 11002 tablet_replica.cc:330] T cc231401bc374df3a18d5e039b36a2e9 P a397a32a5c4c4681951c1de266125f9c: stopping tablet replica
I20260806 06:28:58.966802 10558 tablet_replica.cc:330] T cc231401bc374df3a18d5e039b36a2e9 P e7a2421395c9473e91a329c11588a784: stopping tablet replica
I20260806 06:28:58.966850 10558 raft_consensus.cc:2229] T cc231401bc374df3a18d5e039b36a2e9 P e7a2421395c9473e91a329c11588a784 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 06:28:58.966861 11002 raft_consensus.cc:2229] T cc231401bc374df3a18d5e039b36a2e9 P a397a32a5c4c4681951c1de266125f9c [term 1 LEADER]: Raft consensus shutting down.
I20260806 06:28:58.966861 21554 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHhLhT/build/release/bin/kudu with pid 10324
I20260806 06:28:58.966898 10558 raft_consensus.cc:2258] T cc231401bc374df3a18d5e039b36a2e9 P e7a2421395c9473e91a329c11588a784 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 06:28:58.966964 11002 raft_consensus.cc:2258] T cc231401bc374df3a18d5e039b36a2e9 P a397a32a5c4c4681951c1de266125f9c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 06:28:58.967394 11002 ts_tablet_manager.cc:1905] T cc231401bc374df3a18d5e039b36a2e9 P a397a32a5c4c4681951c1de266125f9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 06:28:58.969028 11002 ts_tablet_manager.cc:1918] T cc231401bc374df3a18d5e039b36a2e9 P a397a32a5c4c4681951c1de266125f9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 06:28:58.969094 11002 log.cc:1196] T cc231401bc374df3a18d5e039b36a2e9 P a397a32a5c4c4681951c1de266125f9c: Deleting WAL directory at /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/minicluster-data/ts-1/wal/wals/cc231401bc374df3a18d5e039b36a2e9
I20260806 06:28:58.969367 11002 ts_tablet_manager.cc:1939] T cc231401bc374df3a18d5e039b36a2e9 P a397a32a5c4c4681951c1de266125f9c: Deleting consensus metadata
I20260806 06:28:58.969777 10231 catalog_manager.cc:4841] TS a397a32a5c4c4681951c1de266125f9c (127.21.12.130:43341): tablet cc231401bc374df3a18d5e039b36a2e9 (table test-table [id=6821954238e54e73b36e296119fa1fb4]) successfully deleted
I20260806 06:28:58.969924 11004 tablet_replica.cc:330] T cc231401bc374df3a18d5e039b36a2e9 P b778fbcc33a740b0876dd34c9fa6829c: stopping tablet replica
I20260806 06:28:58.970011 11004 raft_consensus.cc:2229] T cc231401bc374df3a18d5e039b36a2e9 P b778fbcc33a740b0876dd34c9fa6829c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 06:28:58.970103 11004 raft_consensus.cc:2258] T cc231401bc374df3a18d5e039b36a2e9 P b778fbcc33a740b0876dd34c9fa6829c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 06:28:58.972114 11004 ts_tablet_manager.cc:1905] T cc231401bc374df3a18d5e039b36a2e9 P b778fbcc33a740b0876dd34c9fa6829c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 06:28:58.973587 11004 ts_tablet_manager.cc:1918] T cc231401bc374df3a18d5e039b36a2e9 P b778fbcc33a740b0876dd34c9fa6829c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 06:28:58.973649 11004 log.cc:1196] T cc231401bc374df3a18d5e039b36a2e9 P b778fbcc33a740b0876dd34c9fa6829c: Deleting WAL directory at /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/minicluster-data/ts-2/wal/wals/cc231401bc374df3a18d5e039b36a2e9
I20260806 06:28:58.973912 11004 ts_tablet_manager.cc:1939] T cc231401bc374df3a18d5e039b36a2e9 P b778fbcc33a740b0876dd34c9fa6829c: Deleting consensus metadata
I20260806 06:28:58.974334 10231 catalog_manager.cc:4841] TS b778fbcc33a740b0876dd34c9fa6829c (127.21.12.131:32863): tablet cc231401bc374df3a18d5e039b36a2e9 (table test-table [id=6821954238e54e73b36e296119fa1fb4]) successfully deleted
W20260806 06:28:58.984802 10231 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.12.129:42271 (error 108)
I20260806 06:28:58.985262 21554 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHhLhT/build/release/bin/kudu with pid 10552
I20260806 06:28:58.985271 10966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:28:58.985132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.129:42271 (local address 127.0.0.1:42292)
0806 06:28:58.985189 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0806 06:28:58.985235 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.129:42271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260806 06:28:58.992926 10231 catalog_manager.cc:4584] TS e7a2421395c9473e91a329c11588a784 (127.21.12.129:42271): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet cc231401bc374df3a18d5e039b36a2e9: Network error: Client connection negotiation failed: client connection to 127.21.12.129:42271: connect: Connection refused (error 111)
I20260806 06:28:58.997643 21554 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHhLhT/build/release/bin/kudu with pid 10768
I20260806 06:28:59.006093 21554 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHhLhT/build/release/bin/kudu with pid 10204
I20260806 06:28:59.015770 21554 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHhLhT/build/release/bin/kudu
/tmp/dist-test-taskFHhLhT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.12.190:36809
--webserver_interface=127.21.12.190
--webserver_port=44209
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.190.keytab
--principal=kudu/127.21.12.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.148:41671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.12.190:36809
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 06:28:59.097892 11006 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 06:28:59.098078 11006 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 06:28:59.098099 11006 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 06:28:59.099208 11006 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 06:28:59.099246 11006 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 06:28:59.099257 11006 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 06:28:59.099268 11006 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 06:28:59.099279 11006 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 06:28:59.100684 11006 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.148:41671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.12.190:36809
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.190.keytab
--principal=kudu/127.21.12.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.12.190:36809
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.12.190
--webserver_port=44209
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11006
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 05:11:02 UTC on ccycloud.quasar-hvzbbz.root.comops.site
build id 4569617
I20260806 06:28:59.100868 11006 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 06:28:59.101030 11006 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 06:28:59.105624 11020 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:28:59.105624 11017 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:28:59.105707 11006 server_base.cc:1034] running on GCE node
W20260806 06:28:59.105724 11015 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:28:59.106077 11006 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 06:28:59.106366 11006 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 06:28:59.107501 11006 hybrid_clock.cc:648] HybridClock initialized: now 1785997739107488 us; error 29 us; skew 500 ppm
Aug 06 06:28:59 dist-test-slave-4tdj krb5kdc[9417](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997739, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 06:28:59.109731 11006 init.cc:377] Logged in from keytab as kudu/127.21.12.190@KRBTEST.COM (short username kudu)
I20260806 06:28:59.110138 11006 webserver.cc:460] Webserver started at http://127.21.12.190:44209/ using document root <none> and password file <none>
I20260806 06:28:59.110383 11006 fs_manager.cc:359] Metadata directory not provided
I20260806 06:28:59.110430 11006 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 06:28:59.111661 11006 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260806 06:28:59.112341 11029 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 06:28:59.112479 11006 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260806 06:28:59.112558 11006 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/minicluster-data/master-0/data,/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/minicluster-data/master-0/wal
uuid: "2ad50aad27464833af0c85627af4d871"
format_stamp: "Formatted at 2026-08-06 06:28:57 on dist-test-slave-4tdj"
server_key: "e7f5ee43b1a59a3d790275abcf9c16b2"
server_key_iv: "4be3aafe4a3837f23cfd543aa9b2c504"
server_key_version: "encryptionkey@0"
I20260806 06:28:59.112872 11006 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 06:28:59.119040 11006 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 06:28:59.120587 11006 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 06:28:59.120700 11006 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 06:28:59.124423 11006 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.190:36809
I20260806 06:28:59.124459 11101 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.190:36809 every 8 connection(s)
I20260806 06:28:59.124778 11006 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/minicluster-data/master-0/data/info.pb
I20260806 06:28:59.126319 11102 sys_catalog.cc:263] Verifying existing consensus state
I20260806 06:28:59.126952 11102 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ad50aad27464833af0c85627af4d871: Bootstrap starting.
I20260806 06:28:59.130594 21554 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHhLhT/build/release/bin/kudu as pid 11006
I20260806 06:28:59.130841 21554 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHhLhT/build/release/bin/kudu
/tmp/dist-test-taskFHhLhT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.12.129:42271
--local_ip_for_outbound_sockets=127.21.12.129
--tserver_master_addrs=127.21.12.190:36809
--webserver_port=38883
--webserver_interface=127.21.12.129
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.129.keytab
--principal=kudu/127.21.12.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.148:41671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 06:28:59.135349 11102 log.cc:824] T 00000000000000000000000000000000 P 2ad50aad27464833af0c85627af4d871: Log is configured to *not* fsync() on all Append() calls
I20260806 06:28:59.137497 11102 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ad50aad27464833af0c85627af4d871: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 06:28:59.137754 11102 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ad50aad27464833af0c85627af4d871: Bootstrap complete.
I20260806 06:28:59.139796 11102 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2ad50aad27464833af0c85627af4d871 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ad50aad27464833af0c85627af4d871" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 36809 } }
I20260806 06:28:59.140036 11102 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2ad50aad27464833af0c85627af4d871 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ad50aad27464833af0c85627af4d871, State: Initialized, Role: FOLLOWER
I20260806 06:28:59.140202 11102 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2ad50aad27464833af0c85627af4d871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ad50aad27464833af0c85627af4d871" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 36809 } }
I20260806 06:28:59.140278 11102 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2ad50aad27464833af0c85627af4d871 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 06:28:59.140326 11102 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2ad50aad27464833af0c85627af4d871 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 06:28:59.140372 11102 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2ad50aad27464833af0c85627af4d871 [term 2 FOLLOWER]: Advancing to term 3
I20260806 06:28:59.141311 11102 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2ad50aad27464833af0c85627af4d871 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ad50aad27464833af0c85627af4d871" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 36809 } }
I20260806 06:28:59.141404 11102 leader_election.cc:302] T 00000000000000000000000000000000 P 2ad50aad27464833af0c85627af4d871 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2ad50aad27464833af0c85627af4d871; no voters:
I20260806 06:28:59.141551 11102 leader_election.cc:288] T 00000000000000000000000000000000 P 2ad50aad27464833af0c85627af4d871 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260806 06:28:59.141618 11107 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2ad50aad27464833af0c85627af4d871 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 06:28:59.141789 11102 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2ad50aad27464833af0c85627af4d871 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 06:28:59.141793 11107 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2ad50aad27464833af0c85627af4d871 [term 3 LEADER]: Becoming Leader. State: Replica: 2ad50aad27464833af0c85627af4d871, State: Running, Role: LEADER
I20260806 06:28:59.141954 11107 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2ad50aad27464833af0c85627af4d871 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ad50aad27464833af0c85627af4d871" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 36809 } }
I20260806 06:28:59.142347 11110 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ad50aad27464833af0c85627af4d871 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "2ad50aad27464833af0c85627af4d871" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ad50aad27464833af0c85627af4d871" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 36809 } } }
I20260806 06:28:59.142360 11111 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ad50aad27464833af0c85627af4d871 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2ad50aad27464833af0c85627af4d871. Latest consensus state: current_term: 3 leader_uuid: "2ad50aad27464833af0c85627af4d871" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ad50aad27464833af0c85627af4d871" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 36809 } } }
I20260806 06:28:59.142558 11111 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ad50aad27464833af0c85627af4d871 [sys.catalog]: This master's current role is: LEADER
I20260806 06:28:59.142738 11110 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ad50aad27464833af0c85627af4d871 [sys.catalog]: This master's current role is: LEADER
I20260806 06:28:59.143544 11128 catalog_manager.cc:1259] Loaded cluster ID: 377b98b622474c4ab577dd247b3d558d
I20260806 06:28:59.143591 11128 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 2ad50aad27464833af0c85627af4d871: loading cluster ID for follower catalog manager: success
W20260806 06:28:59.143896 11128 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 2ad50aad27464833af0c85627af4d871: acquiring CA information for follower catalog manager: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
W20260806 06:28:59.143944 11128 catalog_manager.cc:873] Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO: failed to prepare follower catalog manager, will retry
I20260806 06:28:59.143980 11121 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 06:28:59.144320 11121 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 06:28:59.144387 11121 catalog_manager.cc:1259] Loaded cluster ID: 377b98b622474c4ab577dd247b3d558d
I20260806 06:28:59.144415 11121 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260806 06:28:59.144536 11121 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f690580a9ad google::LogMessage::Fail() at ??:0
@ 0x7f690580d88a google::LogMessage::SendToLog() at ??:0
@ 0x7f690580a4de google::LogMessage::Flush() at ??:0
@ 0x7f690580ba19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f690d72b0f1 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7f690d7348b9 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f69076d80d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f69076cd954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f69085316db start_thread at ??:0
@ 0x7f690430861f clone at ??:0
W20260806 06:28:59.223059 11105 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 06:28:59.223248 11105 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 06:28:59.223270 11105 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 06:28:59.224364 11105 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 06:28:59.224402 11105 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 06:28:59.224426 11105 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.12.129
I20260806 06:28:59.225852 11105 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.148:41671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.129.keytab
--principal=kudu/127.21.12.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.12.129:42271
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.12.129
--webserver_port=38883
--tserver_master_addrs=127.21.12.190:36809
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11105
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.12.129
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 05:11:02 UTC on ccycloud.quasar-hvzbbz.root.comops.site
build id 4569617
I20260806 06:28:59.226079 11105 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 06:28:59.226308 11105 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260806 06:28:59.231065 11105 server_base.cc:1034] running on GCE node
W20260806 06:28:59.231036 11148 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:28:59.231046 11150 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:28:59.231238 11152 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:28:59.231495 11105 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 06:28:59.231717 11105 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 06:28:59.232872 11105 hybrid_clock.cc:648] HybridClock initialized: now 1785997739232861 us; error 26 us; skew 500 ppm
Aug 06 06:28:59 dist-test-slave-4tdj krb5kdc[9417](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997739, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 06:28:59.235266 11105 init.cc:377] Logged in from keytab as kudu/127.21.12.129@KRBTEST.COM (short username kudu)
I20260806 06:28:59.235673 11105 webserver.cc:460] Webserver started at http://127.21.12.129:38883/ using document root <none> and password file <none>
I20260806 06:28:59.235894 11105 fs_manager.cc:359] Metadata directory not provided
I20260806 06:28:59.235939 11105 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 06:28:59.237229 11105 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260806 06:28:59.237955 11163 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 06:28:59.238130 11105 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 06:28:59.238238 11105 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/minicluster-data/ts-0/wal
uuid: "e7a2421395c9473e91a329c11588a784"
format_stamp: "Formatted at 2026-08-06 06:28:57 on dist-test-slave-4tdj"
server_key: "e01ae4fd34e538a96a7c18bb93f98cb4"
server_key_iv: "8d4919674b8d1227fbff0693830034c0"
server_key_version: "encryptionkey@0"
I20260806 06:28:59.238588 11105 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 06:28:59.256326 11105 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 06:28:59.258271 11105 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 06:28:59.258420 11105 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 06:28:59.258617 11105 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 06:28:59.259032 11176 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260806 06:28:59.260939 11105 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260806 06:28:59.260983 11105 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260806 06:28:59.261015 11105 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260806 06:28:59.261621 11105 ts_tablet_manager.cc:610] Registered 1 tablets
I20260806 06:28:59.261653 11105 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260806 06:28:59.261742 11176 tablet_bootstrap.cc:492] T cc231401bc374df3a18d5e039b36a2e9 P e7a2421395c9473e91a329c11588a784: Bootstrap starting.
I20260806 06:28:59.268193 11105 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.129:42271
I20260806 06:28:59.268302 11321 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.129:42271 every 8 connection(s)
I20260806 06:28:59.268573 11105 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/minicluster-data/ts-0/data/info.pb
I20260806 06:28:59.272361 11176 log.cc:824] T cc231401bc374df3a18d5e039b36a2e9 P e7a2421395c9473e91a329c11588a784: Log is configured to *not* fsync() on all Append() calls
I20260806 06:28:59.274928 11176 tablet_bootstrap.cc:492] T cc231401bc374df3a18d5e039b36a2e9 P e7a2421395c9473e91a329c11588a784: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 06:28:59.275246 11176 tablet_bootstrap.cc:492] T cc231401bc374df3a18d5e039b36a2e9 P e7a2421395c9473e91a329c11588a784: Bootstrap complete.
I20260806 06:28:59.276290 11176 ts_tablet_manager.cc:1397] T cc231401bc374df3a18d5e039b36a2e9 P e7a2421395c9473e91a329c11588a784: Time spent bootstrapping tablet: real 0.015s user 0.008s sys 0.004s
I20260806 06:28:59.276414 21554 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHhLhT/build/release/bin/kudu as pid 11105
I20260806 06:28:59.276655 21554 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHhLhT/build/release/bin/kudu
/tmp/dist-test-taskFHhLhT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.12.130:43341
--local_ip_for_outbound_sockets=127.21.12.130
--tserver_master_addrs=127.21.12.190:36809
--webserver_port=43723
--webserver_interface=127.21.12.130
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.130.keytab
--principal=kudu/127.21.12.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.148:41671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 06:28:59.277681 11176 raft_consensus.cc:348] T cc231401bc374df3a18d5e039b36a2e9 P e7a2421395c9473e91a329c11588a784 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b778fbcc33a740b0876dd34c9fa6829c" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 32863 } } peers { permanent_uuid: "a397a32a5c4c4681951c1de266125f9c" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 43341 } } peers { permanent_uuid: "e7a2421395c9473e91a329c11588a784" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 42271 } }
I20260806 06:28:59.277999 11176 raft_consensus.cc:729] T cc231401bc374df3a18d5e039b36a2e9 P e7a2421395c9473e91a329c11588a784 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7a2421395c9473e91a329c11588a784, State: Initialized, Role: FOLLOWER
I20260806 06:28:59.278192 11176 consensus_queue.cc:260] T cc231401bc374df3a18d5e039b36a2e9 P e7a2421395c9473e91a329c11588a784 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b778fbcc33a740b0876dd34c9fa6829c" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 32863 } } peers { permanent_uuid: "a397a32a5c4c4681951c1de266125f9c" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 43341 } } peers { permanent_uuid: "e7a2421395c9473e91a329c11588a784" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 42271 } }
I20260806 06:28:59.280177 11176 ts_tablet_manager.cc:1428] T cc231401bc374df3a18d5e039b36a2e9 P e7a2421395c9473e91a329c11588a784: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
W20260806 06:28:59.318500 11327 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 06:28:59.269181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:36809 (local address 127.21.12.129:60661)
0806 06:28:59.269346 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0806 06:28:59.269388 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0806 06:28:59.269825 (+ 437us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:28:59.318267 (+ 48442us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.190:36809: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":76,"threads_started":1}
W20260806 06:28:59.318778 11324 heartbeater.cc:644] Failed to heartbeat to 127.21.12.190:36809 (0 consecutive failures): Network error: Failed to ping master at 127.21.12.190:36809: Client connection negotiation failed: client connection to 127.21.12.190:36809: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260806 06:28:59.319098 11327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:28:59.318986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:36809 (local address 127.21.12.129:52311)
0806 06:28:59.319035 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0806 06:28:59.319076 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.190:36809: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260806 06:28:59.319402 11327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:28:59.319330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:36809 (local address 127.21.12.129:39317)
0806 06:28:59.319364 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0806 06:28:59.319387 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.190:36809: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260806 06:28:59.319545 11324 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260806 06:28:59.359316 11330 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 06:28:59.359508 11330 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 06:28:59.359530 11330 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 06:28:59.360586 11330 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 06:28:59.360618 11330 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 06:28:59.360642 11330 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.12.130
I20260806 06:28:59.361994 11330 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.148:41671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.130.keytab
--principal=kudu/127.21.12.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.12.130:43341
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.12.130
--webserver_port=43723
--tserver_master_addrs=127.21.12.190:36809
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11330
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.12.130
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 05:11:02 UTC on ccycloud.quasar-hvzbbz.root.comops.site
build id 4569617
I20260806 06:28:59.362166 11330 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 06:28:59.362358 11330 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 06:28:59.366958 11348 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:28:59.366957 11345 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:28:59.367038 11343 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:28:59.367283 11330 server_base.cc:1034] running on GCE node
I20260806 06:28:59.367465 11330 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 06:28:59.367664 11330 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 06:28:59.368803 11330 hybrid_clock.cc:648] HybridClock initialized: now 1785997739368791 us; error 28 us; skew 500 ppm
Aug 06 06:28:59 dist-test-slave-4tdj krb5kdc[9417](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997739, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 06:28:59.371199 11330 init.cc:377] Logged in from keytab as kudu/127.21.12.130@KRBTEST.COM (short username kudu)
I20260806 06:28:59.371688 11330 webserver.cc:460] Webserver started at http://127.21.12.130:43723/ using document root <none> and password file <none>
I20260806 06:28:59.371893 11330 fs_manager.cc:359] Metadata directory not provided
I20260806 06:28:59.371940 11330 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 06:28:59.373180 11330 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260806 06:28:59.373736 11356 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 06:28:59.373889 11330 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260806 06:28:59.373980 11330 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/minicluster-data/ts-1/wal
uuid: "a397a32a5c4c4681951c1de266125f9c"
format_stamp: "Formatted at 2026-08-06 06:28:58 on dist-test-slave-4tdj"
server_key: "de85f746c85d5607dfe6f4a622f7693c"
server_key_iv: "0c0d6c91ab3445aad3773bb0edae0806"
server_key_version: "encryptionkey@0"
I20260806 06:28:59.374315 11330 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 06:28:59.411852 11330 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 06:28:59.413964 11330 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 06:28:59.414108 11330 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 06:28:59.414381 11330 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 06:28:59.414698 11330 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 06:28:59.414736 11330 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 06:28:59.414765 11330 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 06:28:59.414788 11330 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 06:28:59.420141 11330 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.130:43341
I20260806 06:28:59.420193 11523 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.130:43341 every 8 connection(s)
I20260806 06:28:59.420524 11330 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/minicluster-data/ts-1/data/info.pb
I20260806 06:28:59.421517 11527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:28:59.421122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:36809 (local address 127.21.12.130:34355)
0806 06:28:59.421275 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0806 06:28:59.421345 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.190:36809: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":82,"threads_started":1}
W20260806 06:28:59.421891 11524 heartbeater.cc:644] Failed to heartbeat to 127.21.12.190:36809 (0 consecutive failures): Network error: Failed to ping master at 127.21.12.190:36809: Client connection negotiation failed: client connection to 127.21.12.190:36809: connect: Connection refused (error 111)
I20260806 06:28:59.422147 11527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:28:59.422058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:36809 (local address 127.21.12.130:56373)
0806 06:28:59.422100 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0806 06:28:59.422126 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.190:36809: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260806 06:28:59.422389 11527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:28:59.422328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:36809 (local address 127.21.12.130:38553)
0806 06:28:59.422352 (+ 24us) negotiation.cc:107] Waiting for socket to connect
0806 06:28:59.422373 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.190:36809: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":13}
W20260806 06:28:59.422485 11524 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260806 06:28:59.424501 21554 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHhLhT/build/release/bin/kudu as pid 11330
I20260806 06:28:59.424705 21554 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHhLhT/build/release/bin/kudu
/tmp/dist-test-taskFHhLhT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.12.131:32863
--local_ip_for_outbound_sockets=127.21.12.131
--tserver_master_addrs=127.21.12.190:36809
--webserver_port=42145
--webserver_interface=127.21.12.131
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.131.keytab
--principal=kudu/127.21.12.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.148:41671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 06:28:59.506630 11529 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 06:28:59.506810 11529 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 06:28:59.506831 11529 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 06:28:59.507893 11529 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 06:28:59.507927 11529 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 06:28:59.507951 11529 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.12.131
I20260806 06:28:59.509326 11529 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.148:41671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.131.keytab
--principal=kudu/127.21.12.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.12.131:32863
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.12.131
--webserver_port=42145
--tserver_master_addrs=127.21.12.190:36809
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11529
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.12.131
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 05:11:02 UTC on ccycloud.quasar-hvzbbz.root.comops.site
build id 4569617
I20260806 06:28:59.509492 11529 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 06:28:59.509666 11529 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 06:28:59.514375 11537 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:28:59.514458 11529 server_base.cc:1034] running on GCE node
W20260806 06:28:59.514396 11539 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:28:59.514384 11541 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:28:59.514812 11529 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 06:28:59.515009 11529 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 06:28:59.516158 11529 hybrid_clock.cc:648] HybridClock initialized: now 1785997739516144 us; error 28 us; skew 500 ppm
Aug 06 06:28:59 dist-test-slave-4tdj krb5kdc[9417](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997739, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 06:28:59.518501 11529 init.cc:377] Logged in from keytab as kudu/127.21.12.131@KRBTEST.COM (short username kudu)
I20260806 06:28:59.518930 11529 webserver.cc:460] Webserver started at http://127.21.12.131:42145/ using document root <none> and password file <none>
I20260806 06:28:59.519138 11529 fs_manager.cc:359] Metadata directory not provided
I20260806 06:28:59.519187 11529 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 06:28:59.520438 11529 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260806 06:28:59.520953 11554 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 06:28:59.521101 11529 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260806 06:28:59.521183 11529 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/minicluster-data/ts-2/wal
uuid: "b778fbcc33a740b0876dd34c9fa6829c"
format_stamp: "Formatted at 2026-08-06 06:28:58 on dist-test-slave-4tdj"
server_key: "c10e6fb753d0a8555df8b25ce0e3e3da"
server_key_iv: "a2995c2452fb6ac189c123a98f246f68"
server_key_version: "encryptionkey@0"
I20260806 06:28:59.521464 11529 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 06:28:59.546792 11529 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 06:28:59.548993 11529 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 06:28:59.549134 11529 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 06:28:59.549374 11529 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 06:28:59.549682 11529 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 06:28:59.549716 11529 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 06:28:59.549746 11529 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 06:28:59.549768 11529 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 06:28:59.555336 11529 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.131:32863
I20260806 06:28:59.555399 11735 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.131:32863 every 8 connection(s)
I20260806 06:28:59.555725 11529 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785997581049705-21554-0/minicluster-data/ts-2/data/info.pb
I20260806 06:28:59.556695 11738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:28:59.556315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:36809 (local address 127.21.12.131:57207)
0806 06:28:59.556462 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0806 06:28:59.556549 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.190:36809: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":77,"threads_started":1}
W20260806 06:28:59.557065 11736 heartbeater.cc:644] Failed to heartbeat to 127.21.12.190:36809 (0 consecutive failures): Network error: Failed to ping master at 127.21.12.190:36809: Client connection negotiation failed: client connection to 127.21.12.190:36809: connect: Connection refused (error 111)
I20260806 06:28:59.557305 11738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:28:59.557228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:36809 (local address 127.21.12.131:59297)
0806 06:28:59.557263 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0806 06:28:59.557288 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.190:36809: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
I20260806 06:28:59.557519 11738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:28:59.557457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:36809 (local address 127.21.12.131:58765)
0806 06:28:59.557487 (+ 30us) negotiation.cc:107] Waiting for socket to connect
0806 06:28:59.557506 (+ 19us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.190:36809: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
W20260806 06:28:59.557605 11736 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260806 06:28:59.560590 21554 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHhLhT/build/release/bin/kudu as pid 11529
I20260806 06:29:00.320138 11740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:29:00.319899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:36809 (local address 127.21.12.129:47159)
0806 06:29:00.320029 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0806 06:29:00.320070 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.190:36809: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":64,"threads_started":1}
I20260806 06:29:00.407889 11741 raft_consensus.cc:482] T cc231401bc374df3a18d5e039b36a2e9 P e7a2421395c9473e91a329c11588a784 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 06:29:00.408026 11741 raft_consensus.cc:504] T cc231401bc374df3a18d5e039b36a2e9 P e7a2421395c9473e91a329c11588a784 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b778fbcc33a740b0876dd34c9fa6829c" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 32863 } } peers { permanent_uuid: "a397a32a5c4c4681951c1de266125f9c" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 43341 } } peers { permanent_uuid: "e7a2421395c9473e91a329c11588a784" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 42271 } }
I20260806 06:29:00.408355 11741 leader_election.cc:288] T cc231401bc374df3a18d5e039b36a2e9 P e7a2421395c9473e91a329c11588a784 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b778fbcc33a740b0876dd34c9fa6829c (127.21.12.131:32863), a397a32a5c4c4681951c1de266125f9c (127.21.12.130:43341)
Aug 06 06:29:00 dist-test-slave-4tdj krb5kdc[9417](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997739, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.129@KRBTEST.COM for kudu/127.21.12.131@KRBTEST.COM
Aug 06 06:29:00 dist-test-slave-4tdj krb5kdc[9417](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997739, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.129@KRBTEST.COM for kudu/127.21.12.130@KRBTEST.COM
I20260806 06:29:00.418558 11740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:29:00.408514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.131:32863 (local address 127.21.12.129:35397)
0806 06:29:00.408551 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0806 06:29:00.408562 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0806 06:29:00.408619 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:29:00.409532 (+ 913us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:29:00.409535 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:29:00.409920 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0806 06:29:00.410098 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:29:00.410104 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:29:00.411225 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:29:00.411225 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:29:00.412000 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:29:00.412002 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:29:00.412536 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:29:00.412537 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:29:00.412642 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:29:00.413140 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 06:29:00.413147 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0806 06:29:00.414735 (+ 1588us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 06:29:00.416759 (+ 2024us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:29:00.416767 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:29:00.416769 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:29:00.417154 (+ 385us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:29:00.417424 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:29:00.417425 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:29:00.417425 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:29:00.417563 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:29:00.417998 (+ 435us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 06:29:00.418004 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 06:29:00.418153 (+ 149us) client_negotiation.cc:769] Sending connection context
0806 06:29:00.418259 (+ 106us) client_negotiation.cc:240] Negotiation successful
0806 06:29:00.418446 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18,"mutex_wait_us":61}
I20260806 06:29:00.418576 11744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:29:00.408646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.129:35397 (local address 127.21.12.131:32863)
0806 06:29:00.408827 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0806 06:29:00.408841 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 06:29:00.408868 (+ 27us) server_negotiation.cc:407] Connection header received
0806 06:29:00.409287 (+ 419us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:29:00.409289 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:29:00.409342 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:29:00.409471 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:29:00.410206 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:29:00.411128 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:29:00.412074 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:29:00.412454 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:29:00.412550 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:29:00.414844 (+ 2294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:29:00.414849 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:29:00.414849 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 06:29:00.414861 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:29:00.416680 (+ 1819us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:29:00.417241 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:29:00.417243 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:29:00.417244 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:29:00.417354 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:29:00.417626 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:29:00.417627 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:29:00.417627 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:29:00.417926 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:29:00.418006 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0806 06:29:00.418340 (+ 334us) server_negotiation.cc:299] Negotiation successful
0806 06:29:00.418474 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":90,"threads_started":1}
I20260806 06:29:00.418593 11748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:29:00.408747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.129:36077 (local address 127.21.12.130:43341)
0806 06:29:00.408878 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0806 06:29:00.408889 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 06:29:00.408914 (+ 25us) server_negotiation.cc:407] Connection header received
0806 06:29:00.409335 (+ 421us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:29:00.409337 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:29:00.409385 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:29:00.409502 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:29:00.410205 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:29:00.411134 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:29:00.412083 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:29:00.412451 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:29:00.412553 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:29:00.415113 (+ 2560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:29:00.415117 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:29:00.415117 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 06:29:00.415129 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:29:00.416708 (+ 1579us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:29:00.417240 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:29:00.417243 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:29:00.417244 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:29:00.417354 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:29:00.417626 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:29:00.417627 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:29:00.417627 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:29:00.417926 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:29:00.418006 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0806 06:29:00.418344 (+ 338us) server_negotiation.cc:299] Negotiation successful
0806 06:29:00.418474 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94,"thread_start_us":54,"threads_started":1}
I20260806 06:29:00.419221 11743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:29:00.408640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.130:43341 (local address 127.21.12.129:36077)
0806 06:29:00.408820 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0806 06:29:00.408833 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 06:29:00.408900 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:29:00.409534 (+ 634us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:29:00.409535 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:29:00.409919 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0806 06:29:00.410099 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:29:00.410104 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:29:00.411232 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:29:00.411232 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:29:00.411999 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:29:00.412001 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:29:00.412534 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:29:00.412535 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:29:00.412642 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:29:00.413183 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 06:29:00.413187 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0806 06:29:00.415021 (+ 1834us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 06:29:00.416776 (+ 1755us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:29:00.416777 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:29:00.416777 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:29:00.417154 (+ 377us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:29:00.417416 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:29:00.417417 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:29:00.417417 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:29:00.417563 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:29:00.417998 (+ 435us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 06:29:00.418004 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 06:29:00.418153 (+ 149us) client_negotiation.cc:769] Sending connection context
0806 06:29:00.418259 (+ 106us) client_negotiation.cc:240] Negotiation successful
0806 06:29:00.418509 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"mutex_wait_us":119,"thread_start_us":79,"threads_started":1}
W20260806 06:29:00.419364 11166 leader_election.cc:341] T cc231401bc374df3a18d5e039b36a2e9 P e7a2421395c9473e91a329c11588a784 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b778fbcc33a740b0876dd34c9fa6829c (127.21.12.131:32863): Not found: Tablet not found: cc231401bc374df3a18d5e039b36a2e9
W20260806 06:29:00.419966 11166 leader_election.cc:341] T cc231401bc374df3a18d5e039b36a2e9 P e7a2421395c9473e91a329c11588a784 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a397a32a5c4c4681951c1de266125f9c (127.21.12.130:43341): Not found: Tablet not found: cc231401bc374df3a18d5e039b36a2e9
I20260806 06:29:00.420011 11166 leader_election.cc:302] T cc231401bc374df3a18d5e039b36a2e9 P e7a2421395c9473e91a329c11588a784 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e7a2421395c9473e91a329c11588a784; no voters: a397a32a5c4c4681951c1de266125f9c, b778fbcc33a740b0876dd34c9fa6829c
I20260806 06:29:00.420218 11741 raft_consensus.cc:2726] T cc231401bc374df3a18d5e039b36a2e9 P e7a2421395c9473e91a329c11588a784 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 06:29:00.422894 11750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:29:00.422730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:36809 (local address 127.21.12.130:45435)
0806 06:29:00.422824 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0806 06:29:00.422867 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.190:36809: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81,"thread_start_us":50,"threads_started":1}
I20260806 06:29:00.558089 11752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:29:00.557894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:36809 (local address 127.21.12.131:45465)
0806 06:29:00.557997 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0806 06:29:00.558052 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.190:36809: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87,"thread_start_us":53,"threads_started":1}
I20260806 06:29:01.320773 11753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:29:01.320575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:36809 (local address 127.21.12.129:55735)
0806 06:29:01.320687 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0806 06:29:01.320724 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.190:36809: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":60,"threads_started":1}
I20260806 06:29:01.423450 11756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:29:01.423232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:36809 (local address 127.21.12.130:34025)
0806 06:29:01.423345 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0806 06:29:01.423380 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.190:36809: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":60,"threads_started":1}
I20260806 06:29:01.558699 11758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:29:01.558478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:36809 (local address 127.21.12.131:46347)
0806 06:29:01.558621 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0806 06:29:01.558660 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.190:36809: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":61,"threads_started":1}
I20260806 06:29:02.321439 11759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:29:02.321225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:36809 (local address 127.21.12.129:42481)
0806 06:29:02.321347 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0806 06:29:02.321387 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.190:36809: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":63,"threads_started":1}
I20260806 06:29:02.424080 11760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:29:02.423899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:36809 (local address 127.21.12.130:34253)
0806 06:29:02.424006 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0806 06:29:02.424039 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.190:36809: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":61,"threads_started":1}
I20260806 06:29:02.472244 11761 raft_consensus.cc:482] T cc231401bc374df3a18d5e039b36a2e9 P e7a2421395c9473e91a329c11588a784 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 06:29:02.472343 11761 raft_consensus.cc:504] T cc231401bc374df3a18d5e039b36a2e9 P e7a2421395c9473e91a329c11588a784 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b778fbcc33a740b0876dd34c9fa6829c" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 32863 } } peers { permanent_uuid: "a397a32a5c4c4681951c1de266125f9c" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 43341 } } peers { permanent_uuid: "e7a2421395c9473e91a329c11588a784" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 42271 } }
I20260806 06:29:02.472470 11761 leader_election.cc:288] T cc231401bc374df3a18d5e039b36a2e9 P e7a2421395c9473e91a329c11588a784 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b778fbcc33a740b0876dd34c9fa6829c (127.21.12.131:32863), a397a32a5c4c4681951c1de266125f9c (127.21.12.130:43341)
W20260806 06:29:02.472895 11166 leader_election.cc:341] T cc231401bc374df3a18d5e039b36a2e9 P e7a2421395c9473e91a329c11588a784 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a397a32a5c4c4681951c1de266125f9c (127.21.12.130:43341): Not found: Tablet not found: cc231401bc374df3a18d5e039b36a2e9
W20260806 06:29:02.472966 11166 leader_election.cc:341] T cc231401bc374df3a18d5e039b36a2e9 P e7a2421395c9473e91a329c11588a784 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b778fbcc33a740b0876dd34c9fa6829c (127.21.12.131:32863): Not found: Tablet not found: cc231401bc374df3a18d5e039b36a2e9
I20260806 06:29:02.472992 11166 leader_election.cc:302] T cc231401bc374df3a18d5e039b36a2e9 P e7a2421395c9473e91a329c11588a784 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e7a2421395c9473e91a329c11588a784; no voters: a397a32a5c4c4681951c1de266125f9c, b778fbcc33a740b0876dd34c9fa6829c
I20260806 06:29:02.473078 11761 raft_consensus.cc:2726] T cc231401bc374df3a18d5e039b36a2e9 P e7a2421395c9473e91a329c11588a784 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 06:29:02.559295 11762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:29:02.559065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:36809 (local address 127.21.12.131:42007)
0806 06:29:02.559212 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0806 06:29:02.559254 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.190:36809: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":78,"threads_started":1}
I20260806 06:29:03.322031 11764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:29:03.321838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:36809 (local address 127.21.12.129:34391)
0806 06:29:03.321948 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0806 06:29:03.321984 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.190:36809: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":59,"threads_started":1}
I20260806 06:29:03.424677 11765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:29:03.424467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:36809 (local address 127.21.12.130:59139)
0806 06:29:03.424599 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0806 06:29:03.424635 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.190:36809: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":80,"threads_started":1}
I20260806 06:29:03.559921 11767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:29:03.559713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:36809 (local address 127.21.12.131:39347)
0806 06:29:03.559847 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0806 06:29:03.559883 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.190:36809: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":80,"threads_started":1}
I20260806 06:29:04.322772 11768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:29:04.322546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:36809 (local address 127.21.12.129:40575)
0806 06:29:04.322683 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0806 06:29:04.322724 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.190:36809: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":61,"threads_started":1}
I20260806 06:29:04.425334 11769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:29:04.425147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:36809 (local address 127.21.12.130:60293)
0806 06:29:04.425264 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0806 06:29:04.425296 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.190:36809: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":70,"threads_started":1}
I20260806 06:29:04.560549 11770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:29:04.560363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:36809 (local address 127.21.12.131:44177)
0806 06:29:04.560478 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0806 06:29:04.560511 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.190:36809: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":62,"threads_started":1}
I20260806 06:29:05.323436 11771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:29:05.323218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:36809 (local address 127.21.12.129:46863)
0806 06:29:05.323348 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0806 06:29:05.323390 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.190:36809: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":63,"threads_started":1}
I20260806 06:29:05.425906 11772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:29:05.425697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:36809 (local address 127.21.12.130:57559)
0806 06:29:05.425831 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0806 06:29:05.425868 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.190:36809: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":81,"threads_started":1}
I20260806 06:29:05.561252 11773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:29:05.561033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:36809 (local address 127.21.12.131:58373)
0806 06:29:05.561170 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0806 06:29:05.561209 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.190:36809: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":65,"threads_started":1}
I20260806 06:29:05.598104 11776 raft_consensus.cc:482] T cc231401bc374df3a18d5e039b36a2e9 P e7a2421395c9473e91a329c11588a784 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 06:29:05.598202 11776 raft_consensus.cc:504] T cc231401bc374df3a18d5e039b36a2e9 P e7a2421395c9473e91a329c11588a784 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b778fbcc33a740b0876dd34c9fa6829c" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 32863 } } peers { permanent_uuid: "a397a32a5c4c4681951c1de266125f9c" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 43341 } } peers { permanent_uuid: "e7a2421395c9473e91a329c11588a784" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 42271 } }
I20260806 06:29:05.598367 11776 leader_election.cc:288] T cc231401bc374df3a18d5e039b36a2e9 P e7a2421395c9473e91a329c11588a784 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b778fbcc33a740b0876dd34c9fa6829c (127.21.12.131:32863), a397a32a5c4c4681951c1de266125f9c (127.21.12.130:43341)
W20260806 06:29:05.598743 11166 leader_election.cc:341] T cc231401bc374df3a18d5e039b36a2e9 P e7a2421395c9473e91a329c11588a784 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b778fbcc33a740b0876dd34c9fa6829c (127.21.12.131:32863): Not found: Tablet not found: cc231401bc374df3a18d5e039b36a2e9
W20260806 06:29:05.598806 11166 leader_election.cc:341] T cc231401bc374df3a18d5e039b36a2e9 P e7a2421395c9473e91a329c11588a784 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a397a32a5c4c4681951c1de266125f9c (127.21.12.130:43341): Not found: Tablet not found: cc231401bc374df3a18d5e039b36a2e9
I20260806 06:29:05.598819 11166 leader_election.cc:302] T cc231401bc374df3a18d5e039b36a2e9 P e7a2421395c9473e91a329c11588a784 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e7a2421395c9473e91a329c11588a784; no voters: a397a32a5c4c4681951c1de266125f9c, b778fbcc33a740b0876dd34c9fa6829c
I20260806 06:29:05.598889 11776 raft_consensus.cc:2726] T cc231401bc374df3a18d5e039b36a2e9 P e7a2421395c9473e91a329c11588a784 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 06:29:06.324131 11778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:29:06.323911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:36809 (local address 127.21.12.129:39353)
0806 06:29:06.324051 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0806 06:29:06.324083 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.190:36809: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":58,"threads_started":1}
I20260806 06:29:06.426545 11779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:29:06.426344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:36809 (local address 127.21.12.130:46339)
0806 06:29:06.426470 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0806 06:29:06.426505 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.190:36809: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":74,"threads_started":1}
I20260806 06:29:06.561857 11781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:29:06.561648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:36809 (local address 127.21.12.131:35439)
0806 06:29:06.561786 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0806 06:29:06.561819 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.190:36809: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":87,"threads_started":1}
I20260806 06:29:07.324967 11782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:29:07.324728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:36809 (local address 127.21.12.129:32809)
0806 06:29:07.324872 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0806 06:29:07.324919 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.190:36809: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":65,"threads_started":1}
I20260806 06:29:07.427258 11783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:29:07.427014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:36809 (local address 127.21.12.130:53299)
0806 06:29:07.427166 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0806 06:29:07.427214 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.190:36809: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":82,"threads_started":1}
I20260806 06:29:07.562534 11785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:29:07.562321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:36809 (local address 127.21.12.131:52137)
0806 06:29:07.562453 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0806 06:29:07.562494 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.190:36809: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":76,"threads_started":1}
I20260806 06:29:08.325639 11788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:29:08.325420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:36809 (local address 127.21.12.129:35851)
0806 06:29:08.325549 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0806 06:29:08.325591 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.190:36809: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":71,"threads_started":1}
I20260806 06:29:08.427881 11789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:29:08.427664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:36809 (local address 127.21.12.130:56341)
0806 06:29:08.427786 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0806 06:29:08.427831 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.190:36809: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":62,"threads_started":1}
I20260806 06:29:08.563163 11790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:29:08.562970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:36809 (local address 127.21.12.131:55589)
0806 06:29:08.563078 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0806 06:29:08.563117 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.190:36809: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":58,"threads_started":1}
I20260806 06:29:09.326272 11791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:29:09.326064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:36809 (local address 127.21.12.129:37059)
0806 06:29:09.326191 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0806 06:29:09.326227 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.190:36809: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":66,"threads_started":1}
I20260806 06:29:09.428486 11792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:29:09.428269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:36809 (local address 127.21.12.130:43823)
0806 06:29:09.428404 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0806 06:29:09.428443 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.190:36809: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":62,"threads_started":1}
I20260806 06:29:09.563704 11794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:29:09.563523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:36809 (local address 127.21.12.131:57353)
0806 06:29:09.563629 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0806 06:29:09.563665 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.190:36809: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":57,"threads_started":1}
I20260806 06:29:09.966168 11795 raft_consensus.cc:482] T cc231401bc374df3a18d5e039b36a2e9 P e7a2421395c9473e91a329c11588a784 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 06:29:09.966266 11795 raft_consensus.cc:504] T cc231401bc374df3a18d5e039b36a2e9 P e7a2421395c9473e91a329c11588a784 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b778fbcc33a740b0876dd34c9fa6829c" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 32863 } } peers { permanent_uuid: "a397a32a5c4c4681951c1de266125f9c" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 43341 } } peers { permanent_uuid: "e7a2421395c9473e91a329c11588a784" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 42271 } }
I20260806 06:29:09.966446 11795 leader_election.cc:288] T cc231401bc374df3a18d5e039b36a2e9 P e7a2421395c9473e91a329c11588a784 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b778fbcc33a740b0876dd34c9fa6829c (127.21.12.131:32863), a397a32a5c4c4681951c1de266125f9c (127.21.12.130:43341)
W20260806 06:29:09.966805 11166 leader_election.cc:341] T cc231401bc374df3a18d5e039b36a2e9 P e7a2421395c9473e91a329c11588a784 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a397a32a5c4c4681951c1de266125f9c (127.21.12.130:43341): Not found: Tablet not found: cc231401bc374df3a18d5e039b36a2e9
W20260806 06:29:09.966895 11166 leader_election.cc:341] T cc231401bc374df3a18d5e039b36a2e9 P e7a2421395c9473e91a329c11588a784 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b778fbcc33a740b0876dd34c9fa6829c (127.21.12.131:32863): Not found: Tablet not found: cc231401bc374df3a18d5e039b36a2e9
I20260806 06:29:09.966926 11166 leader_election.cc:302] T cc231401bc374df3a18d5e039b36a2e9 P e7a2421395c9473e91a329c11588a784 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e7a2421395c9473e91a329c11588a784; no voters: a397a32a5c4c4681951c1de266125f9c, b778fbcc33a740b0876dd34c9fa6829c
I20260806 06:29:09.967011 11795 raft_consensus.cc:2726] T cc231401bc374df3a18d5e039b36a2e9 P e7a2421395c9473e91a329c11588a784 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 06:29:10.326900 11796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:29:10.326707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:36809 (local address 127.21.12.129:38007)
0806 06:29:10.326819 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0806 06:29:10.326850 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.190:36809: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":59,"threads_started":1}
I20260806 06:29:10.429102 11799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:29:10.428890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:36809 (local address 127.21.12.130:51383)
0806 06:29:10.429015 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0806 06:29:10.429056 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.190:36809: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":61,"threads_started":1}
I20260806 06:29:10.564311 11803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:29:10.564079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:36809 (local address 127.21.12.131:50593)
0806 06:29:10.564235 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0806 06:29:10.564270 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.190:36809: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":81,"threads_started":1}
I20260806 06:29:11.327514 11807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:29:11.327279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:36809 (local address 127.21.12.129:47553)
0806 06:29:11.327422 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0806 06:29:11.327465 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.190:36809: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":76,"threads_started":1}
I20260806 06:29:11.429752 11808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:29:11.429505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:36809 (local address 127.21.12.130:50057)
0806 06:29:11.429663 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0806 06:29:11.429700 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.190:36809: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":93,"threads_started":1}
I20260806 06:29:11.564904 11809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:29:11.564718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:36809 (local address 127.21.12.131:55343)
0806 06:29:11.564830 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0806 06:29:11.564863 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.190:36809: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":59,"threads_started":1}
I20260806 06:29:11.876622 11810 raft_consensus.cc:482] T cc231401bc374df3a18d5e039b36a2e9 P e7a2421395c9473e91a329c11588a784 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 06:29:11.876719 11810 raft_consensus.cc:504] T cc231401bc374df3a18d5e039b36a2e9 P e7a2421395c9473e91a329c11588a784 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b778fbcc33a740b0876dd34c9fa6829c" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 32863 } } peers { permanent_uuid: "a397a32a5c4c4681951c1de266125f9c" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 43341 } } peers { permanent_uuid: "e7a2421395c9473e91a329c11588a784" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 42271 } }
I20260806 06:29:11.876852 11810 leader_election.cc:288] T cc231401bc374df3a18d5e039b36a2e9 P e7a2421395c9473e91a329c11588a784 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b778fbcc33a740b0876dd34c9fa6829c (127.21.12.131:32863), a397a32a5c4c4681951c1de266125f9c (127.21.12.130:43341)
W20260806 06:29:11.877228 11166 leader_election.cc:341] T cc231401bc374df3a18d5e039b36a2e9 P e7a2421395c9473e91a329c11588a784 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b778fbcc33a740b0876dd34c9fa6829c (127.21.12.131:32863): Not found: Tablet not found: cc231401bc374df3a18d5e039b36a2e9
W20260806 06:29:11.877290 11166 leader_election.cc:341] T cc231401bc374df3a18d5e039b36a2e9 P e7a2421395c9473e91a329c11588a784 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a397a32a5c4c4681951c1de266125f9c (127.21.12.130:43341): Not found: Tablet not found: cc231401bc374df3a18d5e039b36a2e9
I20260806 06:29:11.877306 11166 leader_election.cc:302] T cc231401bc374df3a18d5e039b36a2e9 P e7a2421395c9473e91a329c11588a784 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e7a2421395c9473e91a329c11588a784; no voters: a397a32a5c4c4681951c1de266125f9c, b778fbcc33a740b0876dd34c9fa6829c
I20260806 06:29:11.877382 11810 raft_consensus.cc:2726] T cc231401bc374df3a18d5e039b36a2e9 P e7a2421395c9473e91a329c11588a784 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 06:29:12.328142 11811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:29:12.327922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:36809 (local address 127.21.12.129:52281)
0806 06:29:12.328057 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0806 06:29:12.328095 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.190:36809: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":60,"threads_started":1}
I20260806 06:29:12.430374 11813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:29:12.430156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:36809 (local address 127.21.12.130:45743)
0806 06:29:12.430278 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0806 06:29:12.430330 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.190:36809: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":56,"threads_started":1}
I20260806 06:29:12.565483 11815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:29:12.565302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:36809 (local address 127.21.12.131:49847)
0806 06:29:12.565409 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0806 06:29:12.565443 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.190:36809: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":60,"threads_started":1}
I20260806 06:29:13.328801 11816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:29:13.328597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:36809 (local address 127.21.12.129:33997)
0806 06:29:13.328718 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0806 06:29:13.328754 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.190:36809: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":68,"threads_started":1}
I20260806 06:29:13.430964 11819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:29:13.430759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:36809 (local address 127.21.12.130:46813)
0806 06:29:13.430880 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0806 06:29:13.430918 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.190:36809: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":61,"threads_started":1}
I20260806 06:29:13.566162 11820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:29:13.565924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:36809 (local address 127.21.12.131:46483)
0806 06:29:13.566078 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0806 06:29:13.566117 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.190:36809: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":97,"threads_started":1}
I20260806 06:29:13.786669 11822 raft_consensus.cc:482] T cc231401bc374df3a18d5e039b36a2e9 P e7a2421395c9473e91a329c11588a784 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 06:29:13.786761 11822 raft_consensus.cc:504] T cc231401bc374df3a18d5e039b36a2e9 P e7a2421395c9473e91a329c11588a784 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b778fbcc33a740b0876dd34c9fa6829c" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 32863 } } peers { permanent_uuid: "a397a32a5c4c4681951c1de266125f9c" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 43341 } } peers { permanent_uuid: "e7a2421395c9473e91a329c11588a784" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 42271 } }
I20260806 06:29:13.786914 11822 leader_election.cc:288] T cc231401bc374df3a18d5e039b36a2e9 P e7a2421395c9473e91a329c11588a784 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b778fbcc33a740b0876dd34c9fa6829c (127.21.12.131:32863), a397a32a5c4c4681951c1de266125f9c (127.21.12.130:43341)
W20260806 06:29:13.787273 11166 leader_election.cc:341] T cc231401bc374df3a18d5e039b36a2e9 P e7a2421395c9473e91a329c11588a784 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a397a32a5c4c4681951c1de266125f9c (127.21.12.130:43341): Not found: Tablet not found: cc231401bc374df3a18d5e039b36a2e9
W20260806 06:29:13.787348 11166 leader_election.cc:341] T cc231401bc374df3a18d5e039b36a2e9 P e7a2421395c9473e91a329c11588a784 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b778fbcc33a740b0876dd34c9fa6829c (127.21.12.131:32863): Not found: Tablet not found: cc231401bc374df3a18d5e039b36a2e9
I20260806 06:29:13.787371 11166 leader_election.cc:302] T cc231401bc374df3a18d5e039b36a2e9 P e7a2421395c9473e91a329c11588a784 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e7a2421395c9473e91a329c11588a784; no voters: a397a32a5c4c4681951c1de266125f9c, b778fbcc33a740b0876dd34c9fa6829c
I20260806 06:29:13.787436 11822 raft_consensus.cc:2726] T cc231401bc374df3a18d5e039b36a2e9 P e7a2421395c9473e91a329c11588a784 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 06:29:14.329523 11823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:29:14.329288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:36809 (local address 127.21.12.129:38987)
0806 06:29:14.329440 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0806 06:29:14.329478 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.190:36809: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":90,"threads_started":1}
I20260806 06:29:14.431643 11825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:29:14.431442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:36809 (local address 127.21.12.130:50297)
0806 06:29:14.431559 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0806 06:29:14.431597 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.190:36809: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":58,"threads_started":1}
I20260806 06:29:14.566991 11826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:29:14.566730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:36809 (local address 127.21.12.131:57695)
0806 06:29:14.566901 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0806 06:29:14.566945 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.190:36809: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":81,"threads_started":1}
I20260806 06:29:14.569098 21554 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHhLhT/build/release/bin/kudu with pid 11105
I20260806 06:29:14.578867 21554 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHhLhT/build/release/bin/kudu with pid 11330
I20260806 06:29:14.585671 21554 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHhLhT/build/release/bin/kudu with pid 11529
2026-08-06T06:29:14Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (18224 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 06:29:14 dist-test-slave-4tdj krb5kdc[11829](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 06:29:14 dist-test-slave-4tdj krb5kdc[11829](info): set up 2 sockets
Aug 06 06:29:14 dist-test-slave-4tdj krb5kdc[11829](info): commencing operation
krb5kdc: starting...
W20260806 06:29:15.700457 21554 mini_kdc.cc:121] Time spent starting KDC: real 1.091s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/krb5kdc/test-user.keytab.
Aug 06 06:29:15 dist-test-slave-4tdj krb5kdc[11829](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997755, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T06:29:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T06:29:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.12.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.12.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.12.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.190.keytab.
Entry for principal kudu/127.21.12.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.190.keytab.
I20260806 06:29:15.814716 21554 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHhLhT/build/release/bin/kudu
/tmp/dist-test-taskFHhLhT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.12.190:35555
--webserver_interface=127.21.12.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.190.keytab
--principal=kudu/127.21.12.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.148:42623
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.12.190:35555
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 06:29:15.894397 11846 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 06:29:15.894593 11846 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 06:29:15.894613 11846 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 06:29:15.895722 11846 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260806 06:29:15.895756 11846 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260806 06:29:15.895766 11846 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260806 06:29:15.895776 11846 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 06:29:15.895792 11846 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 06:29:15.895803 11846 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 06:29:15.895819 11846 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 06:29:15.895830 11846 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 06:29:15.897261 11846 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.148:42623
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.21.12.190:35555
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.190.keytab
--principal=kudu/127.21.12.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.12.190:35555
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.12.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11846
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 05:11:02 UTC on ccycloud.quasar-hvzbbz.root.comops.site
build id 4569617
I20260806 06:29:15.897466 11846 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 06:29:15.897641 11846 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 06:29:15.902532 11855 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:29:15.902567 11854 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:29:15.902597 11846 server_base.cc:1034] running on GCE node
W20260806 06:29:15.902680 11859 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:29:15.903064 11846 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 06:29:15.903333 11846 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 06:29:15.904479 11846 hybrid_clock.cc:648] HybridClock initialized: now 1785997755904463 us; error 29 us; skew 500 ppm
Aug 06 06:29:15 dist-test-slave-4tdj krb5kdc[11829](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997755, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 06:29:15.906919 11846 init.cc:377] Logged in from keytab as kudu/127.21.12.190@KRBTEST.COM (short username kudu)
I20260806 06:29:15.907315 11846 webserver.cc:460] Webserver started at http://127.21.12.190:41507/ using document root <none> and password file <none>
I20260806 06:29:15.907505 11846 fs_manager.cc:359] Metadata directory not provided
I20260806 06:29:15.907549 11846 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 06:29:15.907681 11846 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 06:29:15.908720 11846 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/minicluster-data/master-0/data/instance:
uuid: "b4138ced16fa403fa2a8038460439b20"
format_stamp: "Formatted at 2026-08-06 06:29:15 on dist-test-slave-4tdj"
server_key: "98b1cf0691cf2d63394abfbe6245ad78"
server_key_iv: "5ba1b7f3ade5efc32f2f199f78ee440c"
server_key_version: "encryptionkey@0"
I20260806 06:29:15.909039 11846 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/minicluster-data/master-0/wal/instance:
uuid: "b4138ced16fa403fa2a8038460439b20"
format_stamp: "Formatted at 2026-08-06 06:29:15 on dist-test-slave-4tdj"
server_key: "98b1cf0691cf2d63394abfbe6245ad78"
server_key_iv: "5ba1b7f3ade5efc32f2f199f78ee440c"
server_key_version: "encryptionkey@0"
I20260806 06:29:15.910250 11846 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260806 06:29:15.910990 11874 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 06:29:15.911164 11846 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260806 06:29:15.911249 11846 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/minicluster-data/master-0/data,/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/minicluster-data/master-0/wal
uuid: "b4138ced16fa403fa2a8038460439b20"
format_stamp: "Formatted at 2026-08-06 06:29:15 on dist-test-slave-4tdj"
server_key: "98b1cf0691cf2d63394abfbe6245ad78"
server_key_iv: "5ba1b7f3ade5efc32f2f199f78ee440c"
server_key_version: "encryptionkey@0"
I20260806 06:29:15.911306 11846 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 06:29:15.933364 11846 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 06:29:15.935411 11846 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 06:29:15.935552 11846 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 06:29:15.939466 11846 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.190:35555
I20260806 06:29:15.939504 11943 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.190:35555 every 8 connection(s)
I20260806 06:29:15.939845 11846 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/minicluster-data/master-0/data/info.pb
I20260806 06:29:15.940718 11944 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 06:29:15.943620 11944 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b4138ced16fa403fa2a8038460439b20: Bootstrap starting.
I20260806 06:29:15.944320 11944 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b4138ced16fa403fa2a8038460439b20: Neither blocks nor log segments found. Creating new log.
I20260806 06:29:15.944629 11944 log.cc:824] T 00000000000000000000000000000000 P b4138ced16fa403fa2a8038460439b20: Log is configured to *not* fsync() on all Append() calls
I20260806 06:29:15.945276 11944 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b4138ced16fa403fa2a8038460439b20: No bootstrap required, opened a new log
I20260806 06:29:15.946897 11944 raft_consensus.cc:348] T 00000000000000000000000000000000 P b4138ced16fa403fa2a8038460439b20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4138ced16fa403fa2a8038460439b20" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 35555 } }
I20260806 06:29:15.947033 11944 raft_consensus.cc:374] T 00000000000000000000000000000000 P b4138ced16fa403fa2a8038460439b20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 06:29:15.947067 11944 raft_consensus.cc:729] T 00000000000000000000000000000000 P b4138ced16fa403fa2a8038460439b20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4138ced16fa403fa2a8038460439b20, State: Initialized, Role: FOLLOWER
I20260806 06:29:15.947207 11944 consensus_queue.cc:260] T 00000000000000000000000000000000 P b4138ced16fa403fa2a8038460439b20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4138ced16fa403fa2a8038460439b20" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 35555 } }
I20260806 06:29:15.947285 11944 raft_consensus.cc:388] T 00000000000000000000000000000000 P b4138ced16fa403fa2a8038460439b20 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 06:29:15.947321 11944 raft_consensus.cc:482] T 00000000000000000000000000000000 P b4138ced16fa403fa2a8038460439b20 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 06:29:15.947366 11944 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b4138ced16fa403fa2a8038460439b20 [term 0 FOLLOWER]: Advancing to term 1
I20260806 06:29:15.947849 11944 raft_consensus.cc:504] T 00000000000000000000000000000000 P b4138ced16fa403fa2a8038460439b20 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4138ced16fa403fa2a8038460439b20" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 35555 } }
I20260806 06:29:15.947940 11944 leader_election.cc:302] T 00000000000000000000000000000000 P b4138ced16fa403fa2a8038460439b20 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b4138ced16fa403fa2a8038460439b20; no voters:
I20260806 06:29:15.948103 11944 leader_election.cc:288] T 00000000000000000000000000000000 P b4138ced16fa403fa2a8038460439b20 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 06:29:15.948148 11949 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b4138ced16fa403fa2a8038460439b20 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 06:29:15.948293 11949 raft_consensus.cc:686] T 00000000000000000000000000000000 P b4138ced16fa403fa2a8038460439b20 [term 1 LEADER]: Becoming Leader. State: Replica: b4138ced16fa403fa2a8038460439b20, State: Running, Role: LEADER
I20260806 06:29:15.948354 11944 sys_catalog.cc:564] T 00000000000000000000000000000000 P b4138ced16fa403fa2a8038460439b20 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 06:29:15.948416 11949 consensus_queue.cc:237] T 00000000000000000000000000000000 P b4138ced16fa403fa2a8038460439b20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4138ced16fa403fa2a8038460439b20" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 35555 } }
I20260806 06:29:15.948979 11950 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4138ced16fa403fa2a8038460439b20 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b4138ced16fa403fa2a8038460439b20" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4138ced16fa403fa2a8038460439b20" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 35555 } } }
I20260806 06:29:15.949054 11950 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4138ced16fa403fa2a8038460439b20 [sys.catalog]: This master's current role is: LEADER
I20260806 06:29:15.949121 21554 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHhLhT/build/release/bin/kudu as pid 11846
I20260806 06:29:15.949211 21554 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/minicluster-data/master-0/wal/instance
I20260806 06:29:15.949378 11952 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4138ced16fa403fa2a8038460439b20 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b4138ced16fa403fa2a8038460439b20. Latest consensus state: current_term: 1 leader_uuid: "b4138ced16fa403fa2a8038460439b20" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4138ced16fa403fa2a8038460439b20" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 35555 } } }
I20260806 06:29:15.949430 11952 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4138ced16fa403fa2a8038460439b20 [sys.catalog]: This master's current role is: LEADER
I20260806 06:29:15.949427 21554 external_mini_cluster.cc:1428] Setting key b29be52cbbe5074913609594486f8752
I20260806 06:29:15.949837 11961 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 06:29:15.950362 11961 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 06:29:15.951756 11961 catalog_manager.cc:1347] Generated new cluster ID: 46036382e0414cc0b117e7ffa1720aff
I20260806 06:29:15.951803 11961 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 06 06:29:15 dist-test-slave-4tdj krb5kdc[11829](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997755, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.12.190@KRBTEST.COM
I20260806 06:29:15.957399 11961 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 06:29:15.957904 11961 catalog_manager.cc:1504] Loading token signing keys...
I20260806 06:29:15.960275 11961 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b4138ced16fa403fa2a8038460439b20: Generated new TSK 0
I20260806 06:29:15.960486 11961 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 06:29:15.960680 11972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:29:15.949979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34686 (local address 127.21.12.190:35555)
0806 06:29:15.950153 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0806 06:29:15.950165 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 06:29:15.950195 (+ 30us) server_negotiation.cc:407] Connection header received
0806 06:29:15.950331 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:29:15.950340 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:29:15.950400 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:29:15.950583 (+ 183us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:29:15.951294 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:29:15.952527 (+ 1233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:29:15.953589 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:29:15.954174 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:29:15.954261 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:29:15.957102 (+ 2841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:29:15.957106 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:29:15.957106 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 06:29:15.957122 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:29:15.959105 (+ 1983us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:29:15.959477 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:29:15.959481 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:29:15.959482 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:29:15.959603 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:29:15.959780 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:29:15.959780 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:29:15.959781 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:29:15.960052 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:29:15.960199 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0806 06:29:15.960307 (+ 108us) server_negotiation.cc:299] Negotiation successful
0806 06:29:15.960454 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":72,"threads_started":1}
WARNING: no policy specified for kudu/127.21.12.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.12.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.12.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.129.keytab.
Entry for principal kudu/127.21.12.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.129.keytab.
I20260806 06:29:15.985893 21554 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHhLhT/build/release/bin/kudu
/tmp/dist-test-taskFHhLhT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.12.129:0
--local_ip_for_outbound_sockets=127.21.12.129
--webserver_interface=127.21.12.129
--webserver_port=0
--tserver_master_addrs=127.21.12.190:35555
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.129.keytab
--principal=kudu/127.21.12.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.148:42623
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 06:29:16.065790 11977 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 06:29:16.065992 11977 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 06:29:16.066022 11977 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 06:29:16.067184 11977 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 06:29:16.067225 11977 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 06:29:16.067260 11977 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.12.129
I20260806 06:29:16.068737 11977 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.148:42623
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.129.keytab
--principal=kudu/127.21.12.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.12.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.12.129
--webserver_port=0
--tserver_master_addrs=127.21.12.190:35555
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11977
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.12.129
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 05:11:02 UTC on ccycloud.quasar-hvzbbz.root.comops.site
build id 4569617
I20260806 06:29:16.068943 11977 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 06:29:16.069149 11977 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 06:29:16.073897 11983 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:29:16.073904 11984 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:29:16.073906 11986 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:29:16.074106 11977 server_base.cc:1034] running on GCE node
I20260806 06:29:16.074280 11977 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 06:29:16.074509 11977 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 06:29:16.075644 11977 hybrid_clock.cc:648] HybridClock initialized: now 1785997756075633 us; error 27 us; skew 500 ppm
Aug 06 06:29:16 dist-test-slave-4tdj krb5kdc[11829](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997756, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 06:29:16.077904 11977 init.cc:377] Logged in from keytab as kudu/127.21.12.129@KRBTEST.COM (short username kudu)
I20260806 06:29:16.078284 11977 webserver.cc:460] Webserver started at http://127.21.12.129:39581/ using document root <none> and password file <none>
I20260806 06:29:16.078495 11977 fs_manager.cc:359] Metadata directory not provided
I20260806 06:29:16.078547 11977 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 06:29:16.078670 11977 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 06:29:16.079638 11977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/minicluster-data/ts-0/data/instance:
uuid: "e9deeb824da74ee98f7e811e0c61cfab"
format_stamp: "Formatted at 2026-08-06 06:29:16 on dist-test-slave-4tdj"
server_key: "0f840dd02a07cb7da5a5a90045702fdf"
server_key_iv: "a1916e6fc4f675c3cccfa630297c5803"
server_key_version: "encryptionkey@0"
I20260806 06:29:16.079950 11977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/minicluster-data/ts-0/wal/instance:
uuid: "e9deeb824da74ee98f7e811e0c61cfab"
format_stamp: "Formatted at 2026-08-06 06:29:16 on dist-test-slave-4tdj"
server_key: "0f840dd02a07cb7da5a5a90045702fdf"
server_key_iv: "a1916e6fc4f675c3cccfa630297c5803"
server_key_version: "encryptionkey@0"
I20260806 06:29:16.081112 11977 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260806 06:29:16.081732 11995 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 06:29:16.081883 11977 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260806 06:29:16.081964 11977 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/minicluster-data/ts-0/wal
uuid: "e9deeb824da74ee98f7e811e0c61cfab"
format_stamp: "Formatted at 2026-08-06 06:29:16 on dist-test-slave-4tdj"
server_key: "0f840dd02a07cb7da5a5a90045702fdf"
server_key_iv: "a1916e6fc4f675c3cccfa630297c5803"
server_key_version: "encryptionkey@0"
I20260806 06:29:16.082032 11977 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 06:29:16.092885 11977 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 06:29:16.094503 11977 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 06:29:16.094631 11977 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 06:29:16.094827 11977 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 06:29:16.095111 11977 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 06:29:16.095144 11977 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 06:29:16.095170 11977 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 06:29:16.095188 11977 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 06:29:16.100497 11977 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.129:42233
I20260806 06:29:16.100570 12156 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.129:42233 every 8 connection(s)
I20260806 06:29:16.100819 11977 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/minicluster-data/ts-0/data/info.pb
Aug 06 06:29:16 dist-test-slave-4tdj krb5kdc[11829](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997756, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.129@KRBTEST.COM for kudu/127.21.12.190@KRBTEST.COM
I20260806 06:29:16.109155 11972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:29:16.101419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.129:55187 (local address 127.21.12.190:35555)
0806 06:29:16.101460 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0806 06:29:16.101462 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:29:16.101664 (+ 202us) server_negotiation.cc:407] Connection header received
0806 06:29:16.102144 (+ 480us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:29:16.102145 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:29:16.102158 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:29:16.102208 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:29:16.103135 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:29:16.103636 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:29:16.104560 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:29:16.104764 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:29:16.104817 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:29:16.106833 (+ 2016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:29:16.106835 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:29:16.106835 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 06:29:16.106843 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:29:16.107941 (+ 1098us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:29:16.108286 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:29:16.108287 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:29:16.108288 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:29:16.108346 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:29:16.108552 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:29:16.108553 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:29:16.108553 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:29:16.108664 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:29:16.108721 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0806 06:29:16.108960 (+ 239us) server_negotiation.cc:299] Negotiation successful
0806 06:29:16.109054 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260806 06:29:16.109268 12160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:29:16.101391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:35555 (local address 127.21.12.129:55187)
0806 06:29:16.101548 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0806 06:29:16.101580 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0806 06:29:16.102046 (+ 466us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:29:16.102274 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:29:16.102279 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:29:16.102857 (+ 578us) client_negotiation.cc:189] Negotiated authn=SASL
0806 06:29:16.103035 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:29:16.103040 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:29:16.103718 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:29:16.103719 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:29:16.104481 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:29:16.104483 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:29:16.104831 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:29:16.104832 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:29:16.104954 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:29:16.105427 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 06:29:16.105435 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0806 06:29:16.106748 (+ 1313us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 06:29:16.108012 (+ 1264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:29:16.108019 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:29:16.108021 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:29:16.108221 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:29:16.108401 (+ 180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:29:16.108402 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:29:16.108402 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:29:16.108496 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:29:16.108718 (+ 222us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 06:29:16.108725 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 06:29:16.108847 (+ 122us) client_negotiation.cc:769] Sending connection context
0806 06:29:16.108929 (+ 82us) client_negotiation.cc:240] Negotiation successful
0806 06:29:16.109072 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":83,"threads_started":1}
I20260806 06:29:16.109726 12158 heartbeater.cc:347] Connected to a master server at 127.21.12.190:35555
I20260806 06:29:16.109807 12158 heartbeater.cc:464] Registering TS with master...
I20260806 06:29:16.109938 12158 heartbeater.cc:511] Master 127.21.12.190:35555 requested a full tablet report, sending...
I20260806 06:29:16.110438 11897 ts_manager.cc:194] Registered new tserver with Master: e9deeb824da74ee98f7e811e0c61cfab (127.21.12.129:42233)
I20260806 06:29:16.110658 21554 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHhLhT/build/release/bin/kudu as pid 11977
I20260806 06:29:16.110762 21554 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/minicluster-data/ts-0/wal/instance
I20260806 06:29:16.110957 21554 external_mini_cluster.cc:1428] Setting key 25ae27fa002de1578f8f832a6f5a05f5
I20260806 06:29:16.111285 11897 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.12.129@KRBTEST.COM'} at 127.21.12.129:55187
WARNING: no policy specified for kudu/127.21.12.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.12.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.12.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.130.keytab.
Entry for principal kudu/127.21.12.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.130.keytab.
I20260806 06:29:16.134457 21554 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHhLhT/build/release/bin/kudu
/tmp/dist-test-taskFHhLhT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.12.130:0
--local_ip_for_outbound_sockets=127.21.12.130
--webserver_interface=127.21.12.130
--webserver_port=0
--tserver_master_addrs=127.21.12.190:35555
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.130.keytab
--principal=kudu/127.21.12.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.148:42623
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 06:29:16.213656 12163 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 06:29:16.213842 12163 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 06:29:16.213863 12163 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 06:29:16.215045 12163 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 06:29:16.215085 12163 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 06:29:16.215109 12163 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.12.130
I20260806 06:29:16.216499 12163 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.148:42623
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.130.keytab
--principal=kudu/127.21.12.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.12.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.12.130
--webserver_port=0
--tserver_master_addrs=127.21.12.190:35555
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12163
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.12.130
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 05:11:02 UTC on ccycloud.quasar-hvzbbz.root.comops.site
build id 4569617
I20260806 06:29:16.216678 12163 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 06:29:16.216849 12163 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 06:29:16.221531 12170 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:29:16.221577 12172 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:29:16.221728 12163 server_base.cc:1034] running on GCE node
W20260806 06:29:16.221683 12176 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:29:16.221918 12163 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 06:29:16.222122 12163 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 06:29:16.223264 12163 hybrid_clock.cc:648] HybridClock initialized: now 1785997756223238 us; error 39 us; skew 500 ppm
Aug 06 06:29:16 dist-test-slave-4tdj krb5kdc[11829](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997756, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 06:29:16.225497 12163 init.cc:377] Logged in from keytab as kudu/127.21.12.130@KRBTEST.COM (short username kudu)
I20260806 06:29:16.225899 12163 webserver.cc:460] Webserver started at http://127.21.12.130:42879/ using document root <none> and password file <none>
I20260806 06:29:16.226117 12163 fs_manager.cc:359] Metadata directory not provided
I20260806 06:29:16.226163 12163 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 06:29:16.226282 12163 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 06:29:16.227257 12163 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/minicluster-data/ts-1/data/instance:
uuid: "8c943bdb7a704767808dc5e9d578bc62"
format_stamp: "Formatted at 2026-08-06 06:29:16 on dist-test-slave-4tdj"
server_key: "94402ca88765c1be95e453978be9f69b"
server_key_iv: "86cdf8c29f29fd9a4765778ff5631673"
server_key_version: "encryptionkey@0"
I20260806 06:29:16.227564 12163 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/minicluster-data/ts-1/wal/instance:
uuid: "8c943bdb7a704767808dc5e9d578bc62"
format_stamp: "Formatted at 2026-08-06 06:29:16 on dist-test-slave-4tdj"
server_key: "94402ca88765c1be95e453978be9f69b"
server_key_iv: "86cdf8c29f29fd9a4765778ff5631673"
server_key_version: "encryptionkey@0"
I20260806 06:29:16.228816 12163 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260806 06:29:16.229487 12190 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 06:29:16.229637 12163 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260806 06:29:16.229720 12163 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/minicluster-data/ts-1/wal
uuid: "8c943bdb7a704767808dc5e9d578bc62"
format_stamp: "Formatted at 2026-08-06 06:29:16 on dist-test-slave-4tdj"
server_key: "94402ca88765c1be95e453978be9f69b"
server_key_iv: "86cdf8c29f29fd9a4765778ff5631673"
server_key_version: "encryptionkey@0"
I20260806 06:29:16.229779 12163 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 06:29:16.240561 12163 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 06:29:16.242051 12163 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 06:29:16.242173 12163 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 06:29:16.242379 12163 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 06:29:16.242656 12163 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 06:29:16.242689 12163 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 06:29:16.242714 12163 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 06:29:16.242731 12163 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 06:29:16.248226 12163 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.130:37609
I20260806 06:29:16.248330 12365 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.130:37609 every 8 connection(s)
I20260806 06:29:16.248571 12163 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/minicluster-data/ts-1/data/info.pb
Aug 06 06:29:16 dist-test-slave-4tdj krb5kdc[11829](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997756, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.130@KRBTEST.COM for kudu/127.21.12.190@KRBTEST.COM
I20260806 06:29:16.256815 11972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:29:16.249147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.130:50323 (local address 127.21.12.190:35555)
0806 06:29:16.249187 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0806 06:29:16.249189 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:29:16.249449 (+ 260us) server_negotiation.cc:407] Connection header received
0806 06:29:16.249953 (+ 504us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:29:16.249954 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:29:16.249967 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:29:16.250017 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:29:16.250925 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:29:16.251425 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:29:16.252369 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:29:16.252574 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:29:16.252628 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:29:16.254575 (+ 1947us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:29:16.254577 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:29:16.254577 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 06:29:16.254584 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:29:16.255646 (+ 1062us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:29:16.255984 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:29:16.255985 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:29:16.255985 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:29:16.256037 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:29:16.256240 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:29:16.256241 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:29:16.256242 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:29:16.256349 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:29:16.256404 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0806 06:29:16.256640 (+ 236us) server_negotiation.cc:299] Negotiation successful
0806 06:29:16.256727 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260806 06:29:16.256961 12368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:29:16.249144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:35555 (local address 127.21.12.130:50323)
0806 06:29:16.249298 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0806 06:29:16.249329 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0806 06:29:16.249853 (+ 524us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:29:16.250080 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:29:16.250084 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:29:16.250657 (+ 573us) client_negotiation.cc:189] Negotiated authn=SASL
0806 06:29:16.250833 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:29:16.250837 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:29:16.251505 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:29:16.251507 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:29:16.252284 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:29:16.252287 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:29:16.252646 (+ 359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:29:16.252647 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:29:16.252764 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:29:16.253201 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 06:29:16.253208 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0806 06:29:16.254489 (+ 1281us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 06:29:16.255714 (+ 1225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:29:16.255720 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:29:16.255723 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:29:16.255920 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:29:16.256089 (+ 169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:29:16.256090 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:29:16.256090 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:29:16.256183 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:29:16.256400 (+ 217us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 06:29:16.256407 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 06:29:16.256530 (+ 123us) client_negotiation.cc:769] Sending connection context
0806 06:29:16.256620 (+ 90us) client_negotiation.cc:240] Negotiation successful
0806 06:29:16.256765 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":85,"threads_started":1}
I20260806 06:29:16.257380 12366 heartbeater.cc:347] Connected to a master server at 127.21.12.190:35555
I20260806 06:29:16.257460 12366 heartbeater.cc:464] Registering TS with master...
I20260806 06:29:16.257591 12366 heartbeater.cc:511] Master 127.21.12.190:35555 requested a full tablet report, sending...
I20260806 06:29:16.257903 11895 ts_manager.cc:194] Registered new tserver with Master: 8c943bdb7a704767808dc5e9d578bc62 (127.21.12.130:37609)
I20260806 06:29:16.258239 11895 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.12.130@KRBTEST.COM'} at 127.21.12.130:50323
I20260806 06:29:16.258775 21554 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHhLhT/build/release/bin/kudu as pid 12163
I20260806 06:29:16.258870 21554 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/minicluster-data/ts-1/wal/instance
I20260806 06:29:16.259071 21554 external_mini_cluster.cc:1428] Setting key be6a0682ad4feb94bfce79bda1c3dcb1
WARNING: no policy specified for kudu/127.21.12.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.12.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.12.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.131.keytab.
Entry for principal kudu/127.21.12.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.131.keytab.
I20260806 06:29:16.283195 21554 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHhLhT/build/release/bin/kudu
/tmp/dist-test-taskFHhLhT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.12.131:0
--local_ip_for_outbound_sockets=127.21.12.131
--webserver_interface=127.21.12.131
--webserver_port=0
--tserver_master_addrs=127.21.12.190:35555
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.131.keytab
--principal=kudu/127.21.12.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.148:42623
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 06:29:16.362857 12371 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 06:29:16.363059 12371 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 06:29:16.363090 12371 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 06:29:16.364194 12371 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 06:29:16.364233 12371 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 06:29:16.364267 12371 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.12.131
I20260806 06:29:16.365703 12371 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.148:42623
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.131.keytab
--principal=kudu/127.21.12.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.12.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.12.131
--webserver_port=0
--tserver_master_addrs=127.21.12.190:35555
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12371
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.12.131
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 05:11:02 UTC on ccycloud.quasar-hvzbbz.root.comops.site
build id 4569617
I20260806 06:29:16.365924 12371 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 06:29:16.366120 12371 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 06:29:16.370820 12379 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:29:16.370955 12371 server_base.cc:1034] running on GCE node
W20260806 06:29:16.370827 12382 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:29:16.370853 12377 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:29:16.371306 12371 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 06:29:16.371557 12371 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 06:29:16.372696 12371 hybrid_clock.cc:648] HybridClock initialized: now 1785997756372685 us; error 29 us; skew 500 ppm
Aug 06 06:29:16 dist-test-slave-4tdj krb5kdc[11829](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997756, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 06:29:16.374990 12371 init.cc:377] Logged in from keytab as kudu/127.21.12.131@KRBTEST.COM (short username kudu)
I20260806 06:29:16.375404 12371 webserver.cc:460] Webserver started at http://127.21.12.131:35959/ using document root <none> and password file <none>
I20260806 06:29:16.375577 12371 fs_manager.cc:359] Metadata directory not provided
I20260806 06:29:16.375617 12371 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 06:29:16.375702 12371 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 06:29:16.376618 12371 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/minicluster-data/ts-2/data/instance:
uuid: "ad7b449cdaa54d07b6ff51bf58c98948"
format_stamp: "Formatted at 2026-08-06 06:29:16 on dist-test-slave-4tdj"
server_key: "8b402f2674abfa9a402d4e86c883fcd1"
server_key_iv: "974ce2c7f328631a37b3107550f5261d"
server_key_version: "encryptionkey@0"
I20260806 06:29:16.376904 12371 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/minicluster-data/ts-2/wal/instance:
uuid: "ad7b449cdaa54d07b6ff51bf58c98948"
format_stamp: "Formatted at 2026-08-06 06:29:16 on dist-test-slave-4tdj"
server_key: "8b402f2674abfa9a402d4e86c883fcd1"
server_key_iv: "974ce2c7f328631a37b3107550f5261d"
server_key_version: "encryptionkey@0"
I20260806 06:29:16.378022 12371 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260806 06:29:16.378693 12389 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 06:29:16.378834 12371 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260806 06:29:16.378912 12371 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/minicluster-data/ts-2/wal
uuid: "ad7b449cdaa54d07b6ff51bf58c98948"
format_stamp: "Formatted at 2026-08-06 06:29:16 on dist-test-slave-4tdj"
server_key: "8b402f2674abfa9a402d4e86c883fcd1"
server_key_iv: "974ce2c7f328631a37b3107550f5261d"
server_key_version: "encryptionkey@0"
I20260806 06:29:16.378973 12371 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 06:29:16.390688 12371 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 06:29:16.392357 12371 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 06:29:16.392469 12371 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 06:29:16.392657 12371 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 06:29:16.392930 12371 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 06:29:16.392971 12371 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 06:29:16.392990 12371 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 06:29:16.393013 12371 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 06:29:16.398509 12371 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.131:43851
I20260806 06:29:16.398563 12555 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.131:43851 every 8 connection(s)
I20260806 06:29:16.398828 12371 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/minicluster-data/ts-2/data/info.pb
Aug 06 06:29:16 dist-test-slave-4tdj krb5kdc[11829](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997756, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.131@KRBTEST.COM for kudu/127.21.12.190@KRBTEST.COM
I20260806 06:29:16.406773 11972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:29:16.399425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.131:51605 (local address 127.21.12.190:35555)
0806 06:29:16.399471 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0806 06:29:16.399472 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0806 06:29:16.399669 (+ 197us) server_negotiation.cc:407] Connection header received
0806 06:29:16.400155 (+ 486us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:29:16.400156 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:29:16.400169 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:29:16.400222 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:29:16.401116 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:29:16.401604 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:29:16.402544 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:29:16.402735 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:29:16.402792 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:29:16.404720 (+ 1928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:29:16.404722 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:29:16.404722 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 06:29:16.404729 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:29:16.405747 (+ 1018us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:29:16.406064 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:29:16.406065 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:29:16.406065 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:29:16.406109 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:29:16.406276 (+ 167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:29:16.406276 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:29:16.406277 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:29:16.406397 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:29:16.406448 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0806 06:29:16.406649 (+ 201us) server_negotiation.cc:299] Negotiation successful
0806 06:29:16.406711 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260806 06:29:16.406895 12559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:29:16.399381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:35555 (local address 127.21.12.131:51605)
0806 06:29:16.399557 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0806 06:29:16.399596 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0806 06:29:16.400054 (+ 458us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:29:16.400293 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:29:16.400298 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:29:16.400846 (+ 548us) client_negotiation.cc:189] Negotiated authn=SASL
0806 06:29:16.401023 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:29:16.401027 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:29:16.401684 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:29:16.401685 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:29:16.402459 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:29:16.402461 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:29:16.402805 (+ 344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:29:16.402806 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:29:16.402924 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:29:16.403302 (+ 378us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 06:29:16.403310 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0806 06:29:16.404639 (+ 1329us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 06:29:16.405808 (+ 1169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:29:16.405813 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:29:16.405815 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:29:16.406009 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:29:16.406152 (+ 143us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:29:16.406152 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:29:16.406152 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:29:16.406235 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:29:16.406440 (+ 205us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 06:29:16.406443 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 06:29:16.406545 (+ 102us) client_negotiation.cc:769] Sending connection context
0806 06:29:16.406627 (+ 82us) client_negotiation.cc:240] Negotiation successful
0806 06:29:16.406734 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":91,"threads_started":1}
I20260806 06:29:16.407330 12556 heartbeater.cc:347] Connected to a master server at 127.21.12.190:35555
I20260806 06:29:16.407413 12556 heartbeater.cc:464] Registering TS with master...
I20260806 06:29:16.407572 12556 heartbeater.cc:511] Master 127.21.12.190:35555 requested a full tablet report, sending...
I20260806 06:29:16.407914 11895 ts_manager.cc:194] Registered new tserver with Master: ad7b449cdaa54d07b6ff51bf58c98948 (127.21.12.131:43851)
I20260806 06:29:16.408154 21554 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHhLhT/build/release/bin/kudu as pid 12371
I20260806 06:29:16.408248 21554 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/minicluster-data/ts-2/wal/instance
I20260806 06:29:16.408293 11895 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.12.131@KRBTEST.COM'} at 127.21.12.131:51605
I20260806 06:29:16.408455 21554 external_mini_cluster.cc:1428] Setting key a16a050c5e81d0b06a0764ace2a9d6fb
I20260806 06:29:16.409291 21554 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 06:29:16.415463 11972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:29:16.410551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34702 (local address 127.21.12.190:35555)
0806 06:29:16.410602 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0806 06:29:16.410603 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0806 06:29:16.410711 (+ 108us) server_negotiation.cc:407] Connection header received
0806 06:29:16.410750 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:29:16.410751 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:29:16.410759 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:29:16.410802 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:29:16.411333 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:29:16.411763 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:29:16.412414 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:29:16.412575 (+ 161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:29:16.412613 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:29:16.413556 (+ 943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:29:16.413557 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:29:16.413557 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 06:29:16.413563 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:29:16.414621 (+ 1058us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:29:16.414930 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:29:16.414930 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:29:16.414931 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:29:16.414972 (+ 41us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:29:16.415084 (+ 112us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:29:16.415085 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:29:16.415085 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:29:16.415204 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:29:16.415270 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0806 06:29:16.415356 (+ 86us) server_negotiation.cc:299] Negotiation successful
0806 06:29:16.415408 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260806 06:29:16.416743 11895 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34702:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 06:29:16.418205 11895 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 06:29:16.424017 12568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:29:16.420090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.130:37609 (local address 127.0.0.1:33424)
0806 06:29:16.420652 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0806 06:29:16.420660 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0806 06:29:16.420702 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:29:16.420846 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:29:16.420847 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:29:16.420849 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 06:29:16.420894 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:29:16.420896 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:29:16.421752 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:29:16.421753 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:29:16.423123 (+ 1370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:29:16.423125 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:29:16.423743 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:29:16.423744 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:29:16.423822 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:29:16.423834 (+ 12us) client_negotiation.cc:769] Sending connection context
0806 06:29:16.423897 (+ 63us) client_negotiation.cc:240] Negotiation successful
0806 06:29:16.423945 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":553,"mutex_wait_us":70,"thread_start_us":71,"threads_started":1}
I20260806 06:29:16.424032 12573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:29:16.420275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33424 (local address 127.21.12.130:37609)
0806 06:29:16.420445 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0806 06:29:16.420448 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 06:29:16.420717 (+ 269us) server_negotiation.cc:407] Connection header received
0806 06:29:16.420761 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:29:16.420763 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:29:16.420778 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:29:16.420844 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 06:29:16.420943 (+ 99us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:29:16.421665 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:29:16.423200 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:29:16.423667 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:29:16.423730 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:29:16.423784 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0806 06:29:16.423918 (+ 134us) server_negotiation.cc:299] Negotiation successful
0806 06:29:16.423956 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":94,"threads_started":1}
I20260806 06:29:16.424150 12570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:29:16.420075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.131:43851 (local address 127.0.0.1:36406)
0806 06:29:16.420300 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0806 06:29:16.420341 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0806 06:29:16.420401 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:29:16.420607 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:29:16.420611 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:29:16.420614 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 06:29:16.420699 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:29:16.420701 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:29:16.421894 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:29:16.421895 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:29:16.423074 (+ 1179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:29:16.423076 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:29:16.423946 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:29:16.423947 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:29:16.424009 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:29:16.424013 (+ 4us) client_negotiation.cc:769] Sending connection context
0806 06:29:16.424042 (+ 29us) client_negotiation.cc:240] Negotiation successful
0806 06:29:16.424079 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":197,"mutex_wait_us":1,"thread_start_us":47,"threads_started":1}
I20260806 06:29:16.424309 12572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:29:16.420219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54136 (local address 127.21.12.129:42233)
0806 06:29:16.420482 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0806 06:29:16.420485 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 06:29:16.420564 (+ 79us) server_negotiation.cc:407] Connection header received
0806 06:29:16.420637 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:29:16.420639 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:29:16.420655 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:29:16.420730 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 06:29:16.421094 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:29:16.422071 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:29:16.423179 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:29:16.423851 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:29:16.423916 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:29:16.423982 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0806 06:29:16.424191 (+ 209us) server_negotiation.cc:299] Negotiation successful
0806 06:29:16.424233 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":222,"thread_start_us":72,"threads_started":1}
I20260806 06:29:16.424258 12566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:29:16.420081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.129:42233 (local address 127.0.0.1:54136)
0806 06:29:16.420504 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0806 06:29:16.420512 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0806 06:29:16.420560 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:29:16.420810 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:29:16.420811 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:29:16.420813 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 06:29:16.421014 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:29:16.421016 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:29:16.422155 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:29:16.422156 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:29:16.423096 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:29:16.423099 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:29:16.424011 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:29:16.424012 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:29:16.424063 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:29:16.424065 (+ 2us) client_negotiation.cc:769] Sending connection context
0806 06:29:16.424106 (+ 41us) client_negotiation.cc:240] Negotiation successful
0806 06:29:16.424136 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":413,"mutex_wait_us":3,"thread_start_us":71,"threads_started":1}
I20260806 06:29:16.424551 12571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:29:16.420217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36406 (local address 127.21.12.131:43851)
0806 06:29:16.420383 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0806 06:29:16.420387 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 06:29:16.420399 (+ 12us) server_negotiation.cc:407] Connection header received
0806 06:29:16.420491 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:29:16.420494 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:29:16.420508 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:29:16.420584 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 06:29:16.420777 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:29:16.421802 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:29:16.423152 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:29:16.423848 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:29:16.424235 (+ 387us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:29:16.424293 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0806 06:29:16.424316 (+ 23us) server_negotiation.cc:299] Negotiation successful
0806 06:29:16.424351 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":71,"threads_started":1}
I20260806 06:29:16.425028 12259 tablet_service.cc:1467] Processing CreateTablet for tablet a4c5ca619f794fd88f1c6dc26c86330d (DEFAULT_TABLE table=test-table [id=6ce5df3eea7245b2a8536aa13325415b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 06:29:16.425086 12454 tablet_service.cc:1467] Processing CreateTablet for tablet a4c5ca619f794fd88f1c6dc26c86330d (DEFAULT_TABLE table=test-table [id=6ce5df3eea7245b2a8536aa13325415b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 06:29:16.425349 12259 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4c5ca619f794fd88f1c6dc26c86330d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 06:29:16.425423 12454 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4c5ca619f794fd88f1c6dc26c86330d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 06:29:16.426918 12068 tablet_service.cc:1467] Processing CreateTablet for tablet a4c5ca619f794fd88f1c6dc26c86330d (DEFAULT_TABLE table=test-table [id=6ce5df3eea7245b2a8536aa13325415b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 06:29:16.427243 12068 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4c5ca619f794fd88f1c6dc26c86330d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 06:29:16.427994 12574 tablet_bootstrap.cc:492] T a4c5ca619f794fd88f1c6dc26c86330d P ad7b449cdaa54d07b6ff51bf58c98948: Bootstrap starting.
I20260806 06:29:16.428247 12575 tablet_bootstrap.cc:492] T a4c5ca619f794fd88f1c6dc26c86330d P 8c943bdb7a704767808dc5e9d578bc62: Bootstrap starting.
I20260806 06:29:16.428723 12574 tablet_bootstrap.cc:654] T a4c5ca619f794fd88f1c6dc26c86330d P ad7b449cdaa54d07b6ff51bf58c98948: Neither blocks nor log segments found. Creating new log.
I20260806 06:29:16.428896 12575 tablet_bootstrap.cc:654] T a4c5ca619f794fd88f1c6dc26c86330d P 8c943bdb7a704767808dc5e9d578bc62: Neither blocks nor log segments found. Creating new log.
I20260806 06:29:16.429288 12574 log.cc:824] T a4c5ca619f794fd88f1c6dc26c86330d P ad7b449cdaa54d07b6ff51bf58c98948: Log is configured to *not* fsync() on all Append() calls
I20260806 06:29:16.429447 12575 log.cc:824] T a4c5ca619f794fd88f1c6dc26c86330d P 8c943bdb7a704767808dc5e9d578bc62: Log is configured to *not* fsync() on all Append() calls
I20260806 06:29:16.430130 12575 tablet_bootstrap.cc:492] T a4c5ca619f794fd88f1c6dc26c86330d P 8c943bdb7a704767808dc5e9d578bc62: No bootstrap required, opened a new log
I20260806 06:29:16.430135 12574 tablet_bootstrap.cc:492] T a4c5ca619f794fd88f1c6dc26c86330d P ad7b449cdaa54d07b6ff51bf58c98948: No bootstrap required, opened a new log
I20260806 06:29:16.430199 12574 ts_tablet_manager.cc:1397] T a4c5ca619f794fd88f1c6dc26c86330d P ad7b449cdaa54d07b6ff51bf58c98948: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260806 06:29:16.430198 12575 ts_tablet_manager.cc:1397] T a4c5ca619f794fd88f1c6dc26c86330d P 8c943bdb7a704767808dc5e9d578bc62: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260806 06:29:16.431494 12574 raft_consensus.cc:348] T a4c5ca619f794fd88f1c6dc26c86330d P ad7b449cdaa54d07b6ff51bf58c98948 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad7b449cdaa54d07b6ff51bf58c98948" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 43851 } } peers { permanent_uuid: "e9deeb824da74ee98f7e811e0c61cfab" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 42233 } } peers { permanent_uuid: "8c943bdb7a704767808dc5e9d578bc62" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 37609 } }
I20260806 06:29:16.431497 12575 raft_consensus.cc:348] T a4c5ca619f794fd88f1c6dc26c86330d P 8c943bdb7a704767808dc5e9d578bc62 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad7b449cdaa54d07b6ff51bf58c98948" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 43851 } } peers { permanent_uuid: "e9deeb824da74ee98f7e811e0c61cfab" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 42233 } } peers { permanent_uuid: "8c943bdb7a704767808dc5e9d578bc62" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 37609 } }
I20260806 06:29:16.431615 12574 raft_consensus.cc:374] T a4c5ca619f794fd88f1c6dc26c86330d P ad7b449cdaa54d07b6ff51bf58c98948 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 06:29:16.431624 12575 raft_consensus.cc:374] T a4c5ca619f794fd88f1c6dc26c86330d P 8c943bdb7a704767808dc5e9d578bc62 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 06:29:16.431636 12574 raft_consensus.cc:729] T a4c5ca619f794fd88f1c6dc26c86330d P ad7b449cdaa54d07b6ff51bf58c98948 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad7b449cdaa54d07b6ff51bf58c98948, State: Initialized, Role: FOLLOWER
I20260806 06:29:16.431645 12575 raft_consensus.cc:729] T a4c5ca619f794fd88f1c6dc26c86330d P 8c943bdb7a704767808dc5e9d578bc62 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c943bdb7a704767808dc5e9d578bc62, State: Initialized, Role: FOLLOWER
I20260806 06:29:16.431723 12574 consensus_queue.cc:260] T a4c5ca619f794fd88f1c6dc26c86330d P ad7b449cdaa54d07b6ff51bf58c98948 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad7b449cdaa54d07b6ff51bf58c98948" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 43851 } } peers { permanent_uuid: "e9deeb824da74ee98f7e811e0c61cfab" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 42233 } } peers { permanent_uuid: "8c943bdb7a704767808dc5e9d578bc62" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 37609 } }
I20260806 06:29:16.431726 12575 consensus_queue.cc:260] T a4c5ca619f794fd88f1c6dc26c86330d P 8c943bdb7a704767808dc5e9d578bc62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad7b449cdaa54d07b6ff51bf58c98948" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 43851 } } peers { permanent_uuid: "e9deeb824da74ee98f7e811e0c61cfab" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 42233 } } peers { permanent_uuid: "8c943bdb7a704767808dc5e9d578bc62" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 37609 } }
I20260806 06:29:16.431977 12575 ts_tablet_manager.cc:1428] T a4c5ca619f794fd88f1c6dc26c86330d P 8c943bdb7a704767808dc5e9d578bc62: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260806 06:29:16.431983 12574 ts_tablet_manager.cc:1428] T a4c5ca619f794fd88f1c6dc26c86330d P ad7b449cdaa54d07b6ff51bf58c98948: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260806 06:29:16.432001 12366 heartbeater.cc:503] Master 127.21.12.190:35555 was elected leader, sending a full tablet report...
I20260806 06:29:16.431998 12556 heartbeater.cc:503] Master 127.21.12.190:35555 was elected leader, sending a full tablet report...
I20260806 06:29:16.432152 12581 tablet_bootstrap.cc:492] T a4c5ca619f794fd88f1c6dc26c86330d P e9deeb824da74ee98f7e811e0c61cfab: Bootstrap starting.
I20260806 06:29:16.432802 12581 tablet_bootstrap.cc:654] T a4c5ca619f794fd88f1c6dc26c86330d P e9deeb824da74ee98f7e811e0c61cfab: Neither blocks nor log segments found. Creating new log.
I20260806 06:29:16.433312 12581 log.cc:824] T a4c5ca619f794fd88f1c6dc26c86330d P e9deeb824da74ee98f7e811e0c61cfab: Log is configured to *not* fsync() on all Append() calls
I20260806 06:29:16.433992 12581 tablet_bootstrap.cc:492] T a4c5ca619f794fd88f1c6dc26c86330d P e9deeb824da74ee98f7e811e0c61cfab: No bootstrap required, opened a new log
I20260806 06:29:16.434043 12581 ts_tablet_manager.cc:1397] T a4c5ca619f794fd88f1c6dc26c86330d P e9deeb824da74ee98f7e811e0c61cfab: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260806 06:29:16.435367 12581 raft_consensus.cc:348] T a4c5ca619f794fd88f1c6dc26c86330d P e9deeb824da74ee98f7e811e0c61cfab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad7b449cdaa54d07b6ff51bf58c98948" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 43851 } } peers { permanent_uuid: "e9deeb824da74ee98f7e811e0c61cfab" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 42233 } } peers { permanent_uuid: "8c943bdb7a704767808dc5e9d578bc62" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 37609 } }
I20260806 06:29:16.435490 12581 raft_consensus.cc:374] T a4c5ca619f794fd88f1c6dc26c86330d P e9deeb824da74ee98f7e811e0c61cfab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 06:29:16.435511 12581 raft_consensus.cc:729] T a4c5ca619f794fd88f1c6dc26c86330d P e9deeb824da74ee98f7e811e0c61cfab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9deeb824da74ee98f7e811e0c61cfab, State: Initialized, Role: FOLLOWER
I20260806 06:29:16.435616 12581 consensus_queue.cc:260] T a4c5ca619f794fd88f1c6dc26c86330d P e9deeb824da74ee98f7e811e0c61cfab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad7b449cdaa54d07b6ff51bf58c98948" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 43851 } } peers { permanent_uuid: "e9deeb824da74ee98f7e811e0c61cfab" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 42233 } } peers { permanent_uuid: "8c943bdb7a704767808dc5e9d578bc62" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 37609 } }
I20260806 06:29:16.435833 12581 ts_tablet_manager.cc:1428] T a4c5ca619f794fd88f1c6dc26c86330d P e9deeb824da74ee98f7e811e0c61cfab: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260806 06:29:16.435899 12158 heartbeater.cc:503] Master 127.21.12.190:35555 was elected leader, sending a full tablet report...
W20260806 06:29:16.499081 12367 tablet.cc:2369] T a4c5ca619f794fd88f1c6dc26c86330d P 8c943bdb7a704767808dc5e9d578bc62: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 06:29:16.601513 12159 tablet.cc:2369] T a4c5ca619f794fd88f1c6dc26c86330d P e9deeb824da74ee98f7e811e0c61cfab: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 06:29:16.649343 12557 tablet.cc:2369] T a4c5ca619f794fd88f1c6dc26c86330d P ad7b449cdaa54d07b6ff51bf58c98948: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 06:29:16.892168 12582 raft_consensus.cc:482] T a4c5ca619f794fd88f1c6dc26c86330d P 8c943bdb7a704767808dc5e9d578bc62 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 06:29:16.892302 12582 raft_consensus.cc:504] T a4c5ca619f794fd88f1c6dc26c86330d P 8c943bdb7a704767808dc5e9d578bc62 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad7b449cdaa54d07b6ff51bf58c98948" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 43851 } } peers { permanent_uuid: "e9deeb824da74ee98f7e811e0c61cfab" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 42233 } } peers { permanent_uuid: "8c943bdb7a704767808dc5e9d578bc62" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 37609 } }
I20260806 06:29:16.892632 12582 leader_election.cc:288] T a4c5ca619f794fd88f1c6dc26c86330d P 8c943bdb7a704767808dc5e9d578bc62 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ad7b449cdaa54d07b6ff51bf58c98948 (127.21.12.131:43851), e9deeb824da74ee98f7e811e0c61cfab (127.21.12.129:42233)
I20260806 06:29:16.896567 12571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:29:16.892874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.130:47511 (local address 127.21.12.131:43851)
0806 06:29:16.892921 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0806 06:29:16.892923 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:29:16.893005 (+ 82us) server_negotiation.cc:407] Connection header received
0806 06:29:16.893074 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:29:16.893075 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:29:16.893083 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:29:16.893153 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 06:29:16.893353 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:29:16.894099 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:29:16.895641 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:29:16.896202 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:29:16.896270 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:29:16.896279 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0806 06:29:16.896458 (+ 179us) server_negotiation.cc:299] Negotiation successful
0806 06:29:16.896493 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260806 06:29:16.896567 12572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:29:16.892872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.130:60909 (local address 127.21.12.129:42233)
0806 06:29:16.892921 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0806 06:29:16.892923 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:29:16.893005 (+ 82us) server_negotiation.cc:407] Connection header received
0806 06:29:16.893067 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:29:16.893068 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:29:16.893077 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:29:16.893153 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 06:29:16.893353 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:29:16.894095 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:29:16.895642 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:29:16.896198 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:29:16.896270 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:29:16.896279 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0806 06:29:16.896458 (+ 179us) server_negotiation.cc:299] Negotiation successful
0806 06:29:16.896493 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260806 06:29:16.896622 12591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:29:16.892800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.129:42233 (local address 127.21.12.130:60909)
0806 06:29:16.892928 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0806 06:29:16.892939 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0806 06:29:16.893008 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:29:16.893168 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:29:16.893169 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:29:16.893171 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 06:29:16.893287 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:29:16.893290 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:29:16.894181 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:29:16.894182 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:29:16.895544 (+ 1362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:29:16.895548 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:29:16.896294 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:29:16.896294 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:29:16.896380 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:29:16.896385 (+ 5us) client_negotiation.cc:769] Sending connection context
0806 06:29:16.896449 (+ 64us) client_negotiation.cc:240] Negotiation successful
0806 06:29:16.896492 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"mutex_wait_us":96,"thread_start_us":50,"threads_started":1}
I20260806 06:29:16.896629 12592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:29:16.892800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.131:43851 (local address 127.21.12.130:47511)
0806 06:29:16.892927 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0806 06:29:16.892939 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 06:29:16.893008 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:29:16.893168 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:29:16.893169 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:29:16.893172 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 06:29:16.893287 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:29:16.893290 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:29:16.894180 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:29:16.894181 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:29:16.895544 (+ 1363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:29:16.895548 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:29:16.896292 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:29:16.896293 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:29:16.896382 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:29:16.896386 (+ 4us) client_negotiation.cc:769] Sending connection context
0806 06:29:16.896446 (+ 60us) client_negotiation.cc:240] Negotiation successful
0806 06:29:16.896492 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"mutex_wait_us":67,"thread_start_us":63,"threads_started":1}
I20260806 06:29:16.897006 12095 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4c5ca619f794fd88f1c6dc26c86330d" candidate_uuid: "8c943bdb7a704767808dc5e9d578bc62" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9deeb824da74ee98f7e811e0c61cfab" is_pre_election: true
I20260806 06:29:16.897192 12095 raft_consensus.cc:2454] T a4c5ca619f794fd88f1c6dc26c86330d P e9deeb824da74ee98f7e811e0c61cfab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c943bdb7a704767808dc5e9d578bc62 in term 0.
I20260806 06:29:16.897149 12488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4c5ca619f794fd88f1c6dc26c86330d" candidate_uuid: "8c943bdb7a704767808dc5e9d578bc62" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad7b449cdaa54d07b6ff51bf58c98948" is_pre_election: true
I20260806 06:29:16.897284 12488 raft_consensus.cc:2454] T a4c5ca619f794fd88f1c6dc26c86330d P ad7b449cdaa54d07b6ff51bf58c98948 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c943bdb7a704767808dc5e9d578bc62 in term 0.
I20260806 06:29:16.897440 12193 leader_election.cc:302] T a4c5ca619f794fd88f1c6dc26c86330d P 8c943bdb7a704767808dc5e9d578bc62 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c943bdb7a704767808dc5e9d578bc62, e9deeb824da74ee98f7e811e0c61cfab; no voters:
I20260806 06:29:16.897593 12582 raft_consensus.cc:2781] T a4c5ca619f794fd88f1c6dc26c86330d P 8c943bdb7a704767808dc5e9d578bc62 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 06:29:16.897660 12582 raft_consensus.cc:482] T a4c5ca619f794fd88f1c6dc26c86330d P 8c943bdb7a704767808dc5e9d578bc62 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 06:29:16.897677 12582 raft_consensus.cc:3037] T a4c5ca619f794fd88f1c6dc26c86330d P 8c943bdb7a704767808dc5e9d578bc62 [term 0 FOLLOWER]: Advancing to term 1
I20260806 06:29:16.898404 12582 raft_consensus.cc:504] T a4c5ca619f794fd88f1c6dc26c86330d P 8c943bdb7a704767808dc5e9d578bc62 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad7b449cdaa54d07b6ff51bf58c98948" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 43851 } } peers { permanent_uuid: "e9deeb824da74ee98f7e811e0c61cfab" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 42233 } } peers { permanent_uuid: "8c943bdb7a704767808dc5e9d578bc62" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 37609 } }
I20260806 06:29:16.898528 12582 leader_election.cc:288] T a4c5ca619f794fd88f1c6dc26c86330d P 8c943bdb7a704767808dc5e9d578bc62 [CANDIDATE]: Term 1 election: Requested vote from peers ad7b449cdaa54d07b6ff51bf58c98948 (127.21.12.131:43851), e9deeb824da74ee98f7e811e0c61cfab (127.21.12.129:42233)
I20260806 06:29:16.898687 12488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4c5ca619f794fd88f1c6dc26c86330d" candidate_uuid: "8c943bdb7a704767808dc5e9d578bc62" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad7b449cdaa54d07b6ff51bf58c98948"
I20260806 06:29:16.898702 12095 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4c5ca619f794fd88f1c6dc26c86330d" candidate_uuid: "8c943bdb7a704767808dc5e9d578bc62" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9deeb824da74ee98f7e811e0c61cfab"
I20260806 06:29:16.898767 12095 raft_consensus.cc:3037] T a4c5ca619f794fd88f1c6dc26c86330d P e9deeb824da74ee98f7e811e0c61cfab [term 0 FOLLOWER]: Advancing to term 1
I20260806 06:29:16.898767 12488 raft_consensus.cc:3037] T a4c5ca619f794fd88f1c6dc26c86330d P ad7b449cdaa54d07b6ff51bf58c98948 [term 0 FOLLOWER]: Advancing to term 1
I20260806 06:29:16.899596 12488 raft_consensus.cc:2454] T a4c5ca619f794fd88f1c6dc26c86330d P ad7b449cdaa54d07b6ff51bf58c98948 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c943bdb7a704767808dc5e9d578bc62 in term 1.
I20260806 06:29:16.899626 12095 raft_consensus.cc:2454] T a4c5ca619f794fd88f1c6dc26c86330d P e9deeb824da74ee98f7e811e0c61cfab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c943bdb7a704767808dc5e9d578bc62 in term 1.
I20260806 06:29:16.899786 12193 leader_election.cc:302] T a4c5ca619f794fd88f1c6dc26c86330d P 8c943bdb7a704767808dc5e9d578bc62 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c943bdb7a704767808dc5e9d578bc62, e9deeb824da74ee98f7e811e0c61cfab; no voters:
I20260806 06:29:16.899879 12582 raft_consensus.cc:2781] T a4c5ca619f794fd88f1c6dc26c86330d P 8c943bdb7a704767808dc5e9d578bc62 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 06:29:16.900017 12582 raft_consensus.cc:686] T a4c5ca619f794fd88f1c6dc26c86330d P 8c943bdb7a704767808dc5e9d578bc62 [term 1 LEADER]: Becoming Leader. State: Replica: 8c943bdb7a704767808dc5e9d578bc62, State: Running, Role: LEADER
I20260806 06:29:16.900122 12582 consensus_queue.cc:237] T a4c5ca619f794fd88f1c6dc26c86330d P 8c943bdb7a704767808dc5e9d578bc62 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad7b449cdaa54d07b6ff51bf58c98948" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 43851 } } peers { permanent_uuid: "e9deeb824da74ee98f7e811e0c61cfab" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 42233 } } peers { permanent_uuid: "8c943bdb7a704767808dc5e9d578bc62" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 37609 } }
I20260806 06:29:16.900882 11895 catalog_manager.cc:5495] T a4c5ca619f794fd88f1c6dc26c86330d P 8c943bdb7a704767808dc5e9d578bc62 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8c943bdb7a704767808dc5e9d578bc62 (127.21.12.130). New cstate: current_term: 1 leader_uuid: "8c943bdb7a704767808dc5e9d578bc62" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad7b449cdaa54d07b6ff51bf58c98948" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 43851 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e9deeb824da74ee98f7e811e0c61cfab" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 42233 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c943bdb7a704767808dc5e9d578bc62" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 37609 } health_report { overall_health: HEALTHY } } }
I20260806 06:29:16.972736 12601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:29:16.970515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33426 (local address 127.21.12.130:37609)
0806 06:29:16.970620 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0806 06:29:16.970622 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:29:16.970631 (+ 9us) server_negotiation.cc:407] Connection header received
0806 06:29:16.970672 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:29:16.970673 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:29:16.970692 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:29:16.970762 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 06:29:16.970893 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:29:16.971375 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:29:16.972119 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:29:16.972317 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:29:16.972378 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:29:16.972481 (+ 103us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 06:29:16.972586 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 06:29:16.972645 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0806 06:29:16.972658 (+ 13us) server_negotiation.cc:299] Negotiation successful
0806 06:29:16.972674 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82,"thread_start_us":46,"threads_started":1}
I20260806 06:29:16.974390 12488 raft_consensus.cc:1261] T a4c5ca619f794fd88f1c6dc26c86330d P ad7b449cdaa54d07b6ff51bf58c98948 [term 1 FOLLOWER]: Refusing update from remote peer 8c943bdb7a704767808dc5e9d578bc62: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 06:29:16.974418 12095 raft_consensus.cc:1261] T a4c5ca619f794fd88f1c6dc26c86330d P e9deeb824da74ee98f7e811e0c61cfab [term 1 FOLLOWER]: Refusing update from remote peer 8c943bdb7a704767808dc5e9d578bc62: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 06:29:16.974730 12582 consensus_queue.cc:1035] T a4c5ca619f794fd88f1c6dc26c86330d P 8c943bdb7a704767808dc5e9d578bc62 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad7b449cdaa54d07b6ff51bf58c98948" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 43851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 06:29:16.974870 12593 consensus_queue.cc:1035] T a4c5ca619f794fd88f1c6dc26c86330d P 8c943bdb7a704767808dc5e9d578bc62 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9deeb824da74ee98f7e811e0c61cfab" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 42233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 06:29:16.977751 12611 mvcc.cc:205] Tried to move back new op lower bound from 7315446812564443136 to 7315446812263448576. Current Snapshot: MvccSnapshot[applied={T|T < 7315446812564443136}]
I20260806 06:29:16.978024 12610 mvcc.cc:205] Tried to move back new op lower bound from 7315446812564443136 to 7315446812263448576. Current Snapshot: MvccSnapshot[applied={T|T < 7315446812564443136}]
I20260806 06:29:16.979089 11895 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34702:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 06:29:16.979173 11895 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34702:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 06:29:16.979877 11895 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b4138ced16fa403fa2a8038460439b20: Sending DeleteTablet for 3 replicas of tablet a4c5ca619f794fd88f1c6dc26c86330d
I20260806 06:29:16.980187 12068 tablet_service.cc:1514] Processing DeleteTablet for tablet a4c5ca619f794fd88f1c6dc26c86330d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 06:29:16 UTC) from {username='kudu', principal='kudu/127.21.12.190@KRBTEST.COM'} at 127.0.0.1:54136
I20260806 06:29:16.980226 12454 tablet_service.cc:1514] Processing DeleteTablet for tablet a4c5ca619f794fd88f1c6dc26c86330d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 06:29:16 UTC) from {username='kudu', principal='kudu/127.21.12.190@KRBTEST.COM'} at 127.0.0.1:36406
I20260806 06:29:16.980259 21554 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHhLhT/build/release/bin/kudu with pid 11977
I20260806 06:29:16.980446 12259 tablet_service.cc:1514] Processing DeleteTablet for tablet a4c5ca619f794fd88f1c6dc26c86330d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 06:29:16 UTC) from {username='kudu', principal='kudu/127.21.12.190@KRBTEST.COM'} at 127.0.0.1:33424
I20260806 06:29:16.980674 12621 tablet_replica.cc:330] T a4c5ca619f794fd88f1c6dc26c86330d P ad7b449cdaa54d07b6ff51bf58c98948: stopping tablet replica
I20260806 06:29:16.980765 12621 raft_consensus.cc:2229] T a4c5ca619f794fd88f1c6dc26c86330d P ad7b449cdaa54d07b6ff51bf58c98948 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 06:29:16.980854 12621 raft_consensus.cc:2258] T a4c5ca619f794fd88f1c6dc26c86330d P ad7b449cdaa54d07b6ff51bf58c98948 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 06:29:16.990162 12623 tablet_replica.cc:330] T a4c5ca619f794fd88f1c6dc26c86330d P 8c943bdb7a704767808dc5e9d578bc62: stopping tablet replica
I20260806 06:29:16.990327 12621 ts_tablet_manager.cc:1905] T a4c5ca619f794fd88f1c6dc26c86330d P ad7b449cdaa54d07b6ff51bf58c98948: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 06:29:16.990372 12623 raft_consensus.cc:2229] T a4c5ca619f794fd88f1c6dc26c86330d P 8c943bdb7a704767808dc5e9d578bc62 [term 1 LEADER]: Raft consensus shutting down.
I20260806 06:29:16.990561 12623 raft_consensus.cc:2258] T a4c5ca619f794fd88f1c6dc26c86330d P 8c943bdb7a704767808dc5e9d578bc62 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 06:29:16.990983 12623 ts_tablet_manager.cc:1905] T a4c5ca619f794fd88f1c6dc26c86330d P 8c943bdb7a704767808dc5e9d578bc62: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 06:29:16.992187 12621 ts_tablet_manager.cc:1918] T a4c5ca619f794fd88f1c6dc26c86330d P ad7b449cdaa54d07b6ff51bf58c98948: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 06:29:16.992317 12621 log.cc:1196] T a4c5ca619f794fd88f1c6dc26c86330d P ad7b449cdaa54d07b6ff51bf58c98948: Deleting WAL directory at /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/minicluster-data/ts-2/wal/wals/a4c5ca619f794fd88f1c6dc26c86330d
I20260806 06:29:16.992565 12623 ts_tablet_manager.cc:1918] T a4c5ca619f794fd88f1c6dc26c86330d P 8c943bdb7a704767808dc5e9d578bc62: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 06:29:16.992617 12621 ts_tablet_manager.cc:1939] T a4c5ca619f794fd88f1c6dc26c86330d P ad7b449cdaa54d07b6ff51bf58c98948: Deleting consensus metadata
I20260806 06:29:16.992676 12623 log.cc:1196] T a4c5ca619f794fd88f1c6dc26c86330d P 8c943bdb7a704767808dc5e9d578bc62: Deleting WAL directory at /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/minicluster-data/ts-1/wal/wals/a4c5ca619f794fd88f1c6dc26c86330d
I20260806 06:29:16.992908 12623 ts_tablet_manager.cc:1939] T a4c5ca619f794fd88f1c6dc26c86330d P 8c943bdb7a704767808dc5e9d578bc62: Deleting consensus metadata
I20260806 06:29:16.993139 11879 catalog_manager.cc:4841] TS ad7b449cdaa54d07b6ff51bf58c98948 (127.21.12.131:43851): tablet a4c5ca619f794fd88f1c6dc26c86330d (table test-table [id=6ce5df3eea7245b2a8536aa13325415b]) successfully deleted
I20260806 06:29:16.993340 11875 catalog_manager.cc:4841] TS 8c943bdb7a704767808dc5e9d578bc62 (127.21.12.130:37609): tablet a4c5ca619f794fd88f1c6dc26c86330d (table test-table [id=6ce5df3eea7245b2a8536aa13325415b]) successfully deleted
W20260806 06:29:16.994514 11878 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.12.129:42233 (error 108)
I20260806 06:29:16.994728 21554 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHhLhT/build/release/bin/kudu with pid 12163
I20260806 06:29:17.000474 12626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:29:17.000301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.129:42233 (local address 127.0.0.1:54140)
0806 06:29:17.000399 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0806 06:29:17.000440 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.129:42233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79,"thread_start_us":43,"threads_started":1}
W20260806 06:29:17.000658 11878 catalog_manager.cc:4584] TS e9deeb824da74ee98f7e811e0c61cfab (127.21.12.129:42233): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a4c5ca619f794fd88f1c6dc26c86330d: Network error: Client connection negotiation failed: client connection to 127.21.12.129:42233: connect: Connection refused (error 111)
I20260806 06:29:17.003423 21554 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHhLhT/build/release/bin/kudu with pid 12371
I20260806 06:29:17.017360 21554 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHhLhT/build/release/bin/kudu with pid 11846
I20260806 06:29:37.028155 21554 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHhLhT/build/release/bin/kudu
/tmp/dist-test-taskFHhLhT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.12.190:35555
--webserver_interface=127.21.12.190
--webserver_port=41507
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.190.keytab
--principal=kudu/127.21.12.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.148:42623
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.12.190:35555
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 06:29:37.108366 12628 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 06:29:37.108564 12628 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 06:29:37.108587 12628 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 06:29:37.109678 12628 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260806 06:29:37.109710 12628 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260806 06:29:37.109722 12628 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260806 06:29:37.109732 12628 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 06:29:37.109742 12628 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 06:29:37.109754 12628 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 06:29:37.109764 12628 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 06:29:37.109794 12628 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 06:29:37.111352 12628 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.148:42623
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.21.12.190:35555
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.190.keytab
--principal=kudu/127.21.12.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.12.190:35555
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.12.190
--webserver_port=41507
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12628
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 05:11:02 UTC on ccycloud.quasar-hvzbbz.root.comops.site
build id 4569617
I20260806 06:29:37.111554 12628 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 06:29:37.111733 12628 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260806 06:29:37.120671 12628 server_base.cc:1034] running on GCE node
W20260806 06:29:37.120685 12637 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:29:37.120703 12636 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:29:37.120775 12639 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:29:37.121102 12628 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 06:29:37.121389 12628 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 06:29:37.122531 12628 hybrid_clock.cc:648] HybridClock initialized: now 1785997777122518 us; error 33 us; skew 500 ppm
Aug 06 06:29:37 dist-test-slave-4tdj krb5kdc[11829](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997777, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 06:29:37.128412 12628 init.cc:377] Logged in from keytab as kudu/127.21.12.190@KRBTEST.COM (short username kudu)
I20260806 06:29:37.128855 12628 webserver.cc:460] Webserver started at http://127.21.12.190:41507/ using document root <none> and password file <none>
I20260806 06:29:37.129079 12628 fs_manager.cc:359] Metadata directory not provided
I20260806 06:29:37.129140 12628 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 06:29:37.130463 12628 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260806 06:29:37.131072 12653 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 06:29:37.131219 12628 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260806 06:29:37.131304 12628 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/minicluster-data/master-0/data,/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/minicluster-data/master-0/wal
uuid: "b4138ced16fa403fa2a8038460439b20"
format_stamp: "Formatted at 2026-08-06 06:29:15 on dist-test-slave-4tdj"
server_key: "98b1cf0691cf2d63394abfbe6245ad78"
server_key_iv: "5ba1b7f3ade5efc32f2f199f78ee440c"
server_key_version: "encryptionkey@0"
I20260806 06:29:37.131695 12628 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 06:29:37.139588 12628 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 06:29:37.141142 12628 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 06:29:37.141269 12628 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 06:29:37.145035 12628 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.190:35555
I20260806 06:29:37.145066 12739 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.190:35555 every 8 connection(s)
I20260806 06:29:37.145426 12628 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/minicluster-data/master-0/data/info.pb
I20260806 06:29:37.146876 12740 sys_catalog.cc:263] Verifying existing consensus state
I20260806 06:29:37.147475 12740 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b4138ced16fa403fa2a8038460439b20: Bootstrap starting.
I20260806 06:29:37.153630 21554 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHhLhT/build/release/bin/kudu as pid 12628
I20260806 06:29:37.153980 21554 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHhLhT/build/release/bin/kudu
/tmp/dist-test-taskFHhLhT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.12.129:42233
--local_ip_for_outbound_sockets=127.21.12.129
--tserver_master_addrs=127.21.12.190:35555
--webserver_port=39581
--webserver_interface=127.21.12.129
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.129.keytab
--principal=kudu/127.21.12.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.148:42623
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 06:29:37.156165 12740 log.cc:824] T 00000000000000000000000000000000 P b4138ced16fa403fa2a8038460439b20: Log is configured to *not* fsync() on all Append() calls
I20260806 06:29:37.158005 12740 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b4138ced16fa403fa2a8038460439b20: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 06:29:37.158315 12740 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b4138ced16fa403fa2a8038460439b20: Bootstrap complete.
I20260806 06:29:37.160406 12740 raft_consensus.cc:348] T 00000000000000000000000000000000 P b4138ced16fa403fa2a8038460439b20 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4138ced16fa403fa2a8038460439b20" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 35555 } }
I20260806 06:29:37.160621 12740 raft_consensus.cc:729] T 00000000000000000000000000000000 P b4138ced16fa403fa2a8038460439b20 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4138ced16fa403fa2a8038460439b20, State: Initialized, Role: FOLLOWER
I20260806 06:29:37.160763 12740 consensus_queue.cc:260] T 00000000000000000000000000000000 P b4138ced16fa403fa2a8038460439b20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4138ced16fa403fa2a8038460439b20" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 35555 } }
I20260806 06:29:37.160823 12740 raft_consensus.cc:388] T 00000000000000000000000000000000 P b4138ced16fa403fa2a8038460439b20 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 06:29:37.160868 12740 raft_consensus.cc:482] T 00000000000000000000000000000000 P b4138ced16fa403fa2a8038460439b20 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 06:29:37.160913 12740 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b4138ced16fa403fa2a8038460439b20 [term 1 FOLLOWER]: Advancing to term 2
I20260806 06:29:37.161886 12740 raft_consensus.cc:504] T 00000000000000000000000000000000 P b4138ced16fa403fa2a8038460439b20 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4138ced16fa403fa2a8038460439b20" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 35555 } }
I20260806 06:29:37.161974 12740 leader_election.cc:302] T 00000000000000000000000000000000 P b4138ced16fa403fa2a8038460439b20 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b4138ced16fa403fa2a8038460439b20; no voters:
I20260806 06:29:37.162127 12740 leader_election.cc:288] T 00000000000000000000000000000000 P b4138ced16fa403fa2a8038460439b20 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260806 06:29:37.162176 12746 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b4138ced16fa403fa2a8038460439b20 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 06:29:37.162348 12746 raft_consensus.cc:686] T 00000000000000000000000000000000 P b4138ced16fa403fa2a8038460439b20 [term 2 LEADER]: Becoming Leader. State: Replica: b4138ced16fa403fa2a8038460439b20, State: Running, Role: LEADER
I20260806 06:29:37.162431 12740 sys_catalog.cc:564] T 00000000000000000000000000000000 P b4138ced16fa403fa2a8038460439b20 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 06:29:37.162482 12746 consensus_queue.cc:237] T 00000000000000000000000000000000 P b4138ced16fa403fa2a8038460439b20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4138ced16fa403fa2a8038460439b20" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 35555 } }
I20260806 06:29:37.162791 12747 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4138ced16fa403fa2a8038460439b20 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "b4138ced16fa403fa2a8038460439b20" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4138ced16fa403fa2a8038460439b20" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 35555 } } }
I20260806 06:29:37.162817 12746 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4138ced16fa403fa2a8038460439b20 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b4138ced16fa403fa2a8038460439b20. Latest consensus state: current_term: 2 leader_uuid: "b4138ced16fa403fa2a8038460439b20" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4138ced16fa403fa2a8038460439b20" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 35555 } } }
I20260806 06:29:37.162974 12746 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4138ced16fa403fa2a8038460439b20 [sys.catalog]: This master's current role is: LEADER
I20260806 06:29:37.162963 12747 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4138ced16fa403fa2a8038460439b20 [sys.catalog]: This master's current role is: LEADER
I20260806 06:29:37.163252 12759 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 06:29:37.164316 12759 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 06:29:37.164462 12759 catalog_manager.cc:1259] Loaded cluster ID: 46036382e0414cc0b117e7ffa1720aff
I20260806 06:29:37.164512 12759 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 06:29:37.166004 12759 catalog_manager.cc:1504] Loading token signing keys...
I20260806 06:29:37.166183 12759 catalog_manager.cc:5879] T 00000000000000000000000000000000 P b4138ced16fa403fa2a8038460439b20: Loaded TSK: 0
I20260806 06:29:37.175599 12759 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b4138ced16fa403fa2a8038460439b20: Generated new TSK 1
I20260806 06:29:37.175688 12759 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260806 06:29:37.240036 12743 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 06:29:37.240217 12743 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 06:29:37.240239 12743 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 06:29:37.241302 12743 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 06:29:37.241333 12743 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 06:29:37.241356 12743 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.12.129
I20260806 06:29:37.242805 12743 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.148:42623
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.129.keytab
--principal=kudu/127.21.12.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.12.129:42233
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.12.129
--webserver_port=39581
--tserver_master_addrs=127.21.12.190:35555
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12743
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.12.129
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 05:11:02 UTC on ccycloud.quasar-hvzbbz.root.comops.site
build id 4569617
I20260806 06:29:37.243029 12743 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 06:29:37.243238 12743 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 06:29:37.248045 12778 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:29:37.248059 12779 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:29:37.248327 12743 server_base.cc:1034] running on GCE node
W20260806 06:29:37.248477 12783 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:29:37.248685 12743 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 06:29:37.248943 12743 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 06:29:37.250109 12743 hybrid_clock.cc:648] HybridClock initialized: now 1785997777250096 us; error 28 us; skew 500 ppm
Aug 06 06:29:37 dist-test-slave-4tdj krb5kdc[11829](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997777, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 06:29:37.253018 12743 init.cc:377] Logged in from keytab as kudu/127.21.12.129@KRBTEST.COM (short username kudu)
I20260806 06:29:37.253542 12743 webserver.cc:460] Webserver started at http://127.21.12.129:39581/ using document root <none> and password file <none>
I20260806 06:29:37.253791 12743 fs_manager.cc:359] Metadata directory not provided
I20260806 06:29:37.253876 12743 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 06:29:37.255577 12743 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260806 06:29:37.256316 12799 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 06:29:37.256507 12743 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 06:29:37.256596 12743 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/minicluster-data/ts-0/wal
uuid: "e9deeb824da74ee98f7e811e0c61cfab"
format_stamp: "Formatted at 2026-08-06 06:29:16 on dist-test-slave-4tdj"
server_key: "0f840dd02a07cb7da5a5a90045702fdf"
server_key_iv: "a1916e6fc4f675c3cccfa630297c5803"
server_key_version: "encryptionkey@0"
I20260806 06:29:37.256914 12743 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 06:29:37.272295 12743 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 06:29:37.274256 12743 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 06:29:37.274411 12743 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 06:29:37.274624 12743 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 06:29:37.275041 12810 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260806 06:29:37.276899 12743 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260806 06:29:37.276957 12743 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260806 06:29:37.276998 12743 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260806 06:29:37.277663 12743 ts_tablet_manager.cc:610] Registered 1 tablets
I20260806 06:29:37.277702 12743 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260806 06:29:37.277794 12810 tablet_bootstrap.cc:492] T a4c5ca619f794fd88f1c6dc26c86330d P e9deeb824da74ee98f7e811e0c61cfab: Bootstrap starting.
I20260806 06:29:37.284209 12743 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.129:42233
I20260806 06:29:37.284324 12972 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.129:42233 every 8 connection(s)
I20260806 06:29:37.284605 12743 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/minicluster-data/ts-0/data/info.pb
I20260806 06:29:37.288259 12810 log.cc:824] T a4c5ca619f794fd88f1c6dc26c86330d P e9deeb824da74ee98f7e811e0c61cfab: Log is configured to *not* fsync() on all Append() calls
I20260806 06:29:37.289891 21554 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHhLhT/build/release/bin/kudu as pid 12743
I20260806 06:29:37.290165 21554 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHhLhT/build/release/bin/kudu
/tmp/dist-test-taskFHhLhT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.12.130:37609
--local_ip_for_outbound_sockets=127.21.12.130
--tserver_master_addrs=127.21.12.190:35555
--webserver_port=42879
--webserver_interface=127.21.12.130
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.130.keytab
--principal=kudu/127.21.12.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.148:42623
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 06:29:37.290817 12810 tablet_bootstrap.cc:492] T a4c5ca619f794fd88f1c6dc26c86330d P e9deeb824da74ee98f7e811e0c61cfab: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 06:29:37.291107 12810 tablet_bootstrap.cc:492] T a4c5ca619f794fd88f1c6dc26c86330d P e9deeb824da74ee98f7e811e0c61cfab: Bootstrap complete.
I20260806 06:29:37.292102 12810 ts_tablet_manager.cc:1397] T a4c5ca619f794fd88f1c6dc26c86330d P e9deeb824da74ee98f7e811e0c61cfab: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.008s
I20260806 06:29:37.293393 12810 raft_consensus.cc:348] T a4c5ca619f794fd88f1c6dc26c86330d P e9deeb824da74ee98f7e811e0c61cfab [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad7b449cdaa54d07b6ff51bf58c98948" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 43851 } } peers { permanent_uuid: "e9deeb824da74ee98f7e811e0c61cfab" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 42233 } } peers { permanent_uuid: "8c943bdb7a704767808dc5e9d578bc62" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 37609 } }
I20260806 06:29:37.293699 12810 raft_consensus.cc:729] T a4c5ca619f794fd88f1c6dc26c86330d P e9deeb824da74ee98f7e811e0c61cfab [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9deeb824da74ee98f7e811e0c61cfab, State: Initialized, Role: FOLLOWER
I20260806 06:29:37.293850 12810 consensus_queue.cc:260] T a4c5ca619f794fd88f1c6dc26c86330d P e9deeb824da74ee98f7e811e0c61cfab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad7b449cdaa54d07b6ff51bf58c98948" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 43851 } } peers { permanent_uuid: "e9deeb824da74ee98f7e811e0c61cfab" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 42233 } } peers { permanent_uuid: "8c943bdb7a704767808dc5e9d578bc62" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 37609 } }
I20260806 06:29:37.294122 12810 ts_tablet_manager.cc:1428] T a4c5ca619f794fd88f1c6dc26c86330d P e9deeb824da74ee98f7e811e0c61cfab: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
Aug 06 06:29:37 dist-test-slave-4tdj krb5kdc[11829](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997777, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.129@KRBTEST.COM for kudu/127.21.12.190@KRBTEST.COM
I20260806 06:29:37.299970 12976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:29:37.285391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.129:52805 (local address 127.21.12.190:35555)
0806 06:29:37.288910 (+ 3519us) server_negotiation.cc:206] Beginning negotiation
0806 06:29:37.288917 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 06:29:37.288950 (+ 33us) server_negotiation.cc:407] Connection header received
0806 06:29:37.291180 (+ 2230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:29:37.291188 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:29:37.291248 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:29:37.291380 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:29:37.292172 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:29:37.293180 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:29:37.294368 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:29:37.294760 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:29:37.294855 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:29:37.296913 (+ 2058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:29:37.296917 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:29:37.296918 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 06:29:37.296930 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:29:37.298399 (+ 1469us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:29:37.298772 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:29:37.298775 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:29:37.298776 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:29:37.298867 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:29:37.299100 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:29:37.299101 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:29:37.299101 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:29:37.299313 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:29:37.299387 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0806 06:29:37.299668 (+ 281us) server_negotiation.cc:299] Negotiation successful
0806 06:29:37.299791 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3474,"thread_start_us":80,"threads_started":1}
I20260806 06:29:37.299983 12975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:29:37.285317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:35555 (local address 127.21.12.129:52805)
0806 06:29:37.288560 (+ 3243us) negotiation.cc:107] Waiting for socket to connect
0806 06:29:37.288604 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0806 06:29:37.290930 (+ 2326us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:29:37.291448 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:29:37.291453 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:29:37.291807 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0806 06:29:37.292051 (+ 244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:29:37.292056 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:29:37.293282 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:29:37.293283 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:29:37.294276 (+ 993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:29:37.294279 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:29:37.294838 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:29:37.294839 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:29:37.294971 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:29:37.295489 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 06:29:37.295497 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0806 06:29:37.296817 (+ 1320us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 06:29:37.298468 (+ 1651us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:29:37.298475 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:29:37.298477 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:29:37.298699 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:29:37.298926 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:29:37.298926 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:29:37.298927 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:29:37.299040 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:29:37.299377 (+ 337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 06:29:37.299383 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 06:29:37.299516 (+ 133us) client_negotiation.cc:769] Sending connection context
0806 06:29:37.299603 (+ 87us) client_negotiation.cc:240] Negotiation successful
0806 06:29:37.299777 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3201,"thread_start_us":93,"threads_started":1}
I20260806 06:29:37.300652 12973 heartbeater.cc:347] Connected to a master server at 127.21.12.190:35555
I20260806 06:29:37.300757 12973 heartbeater.cc:464] Registering TS with master...
I20260806 06:29:37.300940 12973 heartbeater.cc:511] Master 127.21.12.190:35555 requested a full tablet report, sending...
I20260806 06:29:37.301677 12680 ts_manager.cc:194] Registered new tserver with Master: e9deeb824da74ee98f7e811e0c61cfab (127.21.12.129:42233)
I20260806 06:29:37.302659 12680 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.12.129@KRBTEST.COM'} at 127.21.12.129:52805
I20260806 06:29:37.303558 12973 heartbeater.cc:503] Master 127.21.12.190:35555 was elected leader, sending a full tablet report...
Aug 06 06:29:37 dist-test-slave-4tdj krb5kdc[11829](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997777, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.190@KRBTEST.COM for kudu/127.21.12.129@KRBTEST.COM
I20260806 06:29:37.309823 12982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:29:37.302244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.129:42233 (local address 127.0.0.1:58042)
0806 06:29:37.302463 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0806 06:29:37.302491 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0806 06:29:37.302547 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:29:37.303138 (+ 591us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:29:37.303143 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:29:37.303366 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0806 06:29:37.303405 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:29:37.303407 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:29:37.304354 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:29:37.304355 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:29:37.304905 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:29:37.304907 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:29:37.305365 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:29:37.305366 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:29:37.305434 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:29:37.305821 (+ 387us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 06:29:37.305825 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0806 06:29:37.307274 (+ 1449us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 06:29:37.308740 (+ 1466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:29:37.308746 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:29:37.308748 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:29:37.308932 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:29:37.309115 (+ 183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:29:37.309115 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:29:37.309116 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:29:37.309174 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:29:37.309482 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 06:29:37.309489 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 06:29:37.309538 (+ 49us) client_negotiation.cc:769] Sending connection context
0806 06:29:37.309609 (+ 71us) client_negotiation.cc:240] Negotiation successful
0806 06:29:37.309723 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":100,"threads_started":1}
I20260806 06:29:37.309832 12983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:29:37.302341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58042 (local address 127.21.12.129:42233)
0806 06:29:37.302943 (+ 602us) server_negotiation.cc:206] Beginning negotiation
0806 06:29:37.302948 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 06:29:37.302957 (+ 9us) server_negotiation.cc:407] Connection header received
0806 06:29:37.302997 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:29:37.302999 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:29:37.303013 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:29:37.303084 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:29:37.303488 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:29:37.304288 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:29:37.304979 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:29:37.305296 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:29:37.305352 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:29:37.307351 (+ 1999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:29:37.307359 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:29:37.307360 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 06:29:37.307366 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:29:37.308664 (+ 1298us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:29:37.309008 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:29:37.309010 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:29:37.309010 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:29:37.309060 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:29:37.309221 (+ 161us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:29:37.309222 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:29:37.309222 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:29:37.309410 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:29:37.309485 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0806 06:29:37.309658 (+ 173us) server_negotiation.cc:299] Negotiation successful
0806 06:29:37.309739 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":571,"thread_start_us":74,"threads_started":1}
I20260806 06:29:37.310460 12877 tablet_service.cc:1514] Processing DeleteTablet for tablet a4c5ca619f794fd88f1c6dc26c86330d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 06:29:16 UTC) from {username='kudu', principal='kudu/127.21.12.190@KRBTEST.COM'} at 127.0.0.1:58042
I20260806 06:29:37.310995 12986 tablet_replica.cc:330] stopping tablet replica
I20260806 06:29:37.311108 12986 raft_consensus.cc:2229] T a4c5ca619f794fd88f1c6dc26c86330d P e9deeb824da74ee98f7e811e0c61cfab [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 06:29:37.311175 12986 raft_consensus.cc:2258] T a4c5ca619f794fd88f1c6dc26c86330d P e9deeb824da74ee98f7e811e0c61cfab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 06:29:37.311473 12986 ts_tablet_manager.cc:1905] T a4c5ca619f794fd88f1c6dc26c86330d P e9deeb824da74ee98f7e811e0c61cfab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 06:29:37.313896 12986 ts_tablet_manager.cc:1918] T a4c5ca619f794fd88f1c6dc26c86330d P e9deeb824da74ee98f7e811e0c61cfab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 06:29:37.313971 12986 log.cc:1196] T a4c5ca619f794fd88f1c6dc26c86330d P e9deeb824da74ee98f7e811e0c61cfab: Deleting WAL directory at /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/minicluster-data/ts-0/wal/wals/a4c5ca619f794fd88f1c6dc26c86330d
I20260806 06:29:37.314210 12986 ts_tablet_manager.cc:1939] T a4c5ca619f794fd88f1c6dc26c86330d P e9deeb824da74ee98f7e811e0c61cfab: Deleting consensus metadata
I20260806 06:29:37.314713 12657 catalog_manager.cc:4841] TS e9deeb824da74ee98f7e811e0c61cfab (127.21.12.129:42233): tablet a4c5ca619f794fd88f1c6dc26c86330d (table test-table [id=6ce5df3eea7245b2a8536aa13325415b]) successfully deleted
W20260806 06:29:37.374804 12978 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 06:29:37.375010 12978 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 06:29:37.375034 12978 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 06:29:37.376139 12978 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 06:29:37.376178 12978 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 06:29:37.376202 12978 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.12.130
I20260806 06:29:37.377573 12978 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.148:42623
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.130.keytab
--principal=kudu/127.21.12.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.12.130:37609
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.12.130
--webserver_port=42879
--tserver_master_addrs=127.21.12.190:35555
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12978
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.12.130
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 05:11:02 UTC on ccycloud.quasar-hvzbbz.root.comops.site
build id 4569617
I20260806 06:29:37.377776 12978 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 06:29:37.377995 12978 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 06:29:37.382771 12994 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:29:37.382911 12978 server_base.cc:1034] running on GCE node
W20260806 06:29:37.382751 12993 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:29:37.382866 12997 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:29:37.383217 12978 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 06:29:37.383435 12978 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 06:29:37.384609 12978 hybrid_clock.cc:648] HybridClock initialized: now 1785997777384598 us; error 27 us; skew 500 ppm
Aug 06 06:29:37 dist-test-slave-4tdj krb5kdc[11829](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997777, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 06:29:37.387104 12978 init.cc:377] Logged in from keytab as kudu/127.21.12.130@KRBTEST.COM (short username kudu)
I20260806 06:29:37.387537 12978 webserver.cc:460] Webserver started at http://127.21.12.130:42879/ using document root <none> and password file <none>
I20260806 06:29:37.387734 12978 fs_manager.cc:359] Metadata directory not provided
I20260806 06:29:37.387792 12978 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 06:29:37.389065 12978 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260806 06:29:37.389631 13011 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 06:29:37.389775 12978 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260806 06:29:37.389864 12978 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/minicluster-data/ts-1/wal
uuid: "8c943bdb7a704767808dc5e9d578bc62"
format_stamp: "Formatted at 2026-08-06 06:29:16 on dist-test-slave-4tdj"
server_key: "94402ca88765c1be95e453978be9f69b"
server_key_iv: "86cdf8c29f29fd9a4765778ff5631673"
server_key_version: "encryptionkey@0"
I20260806 06:29:37.390177 12978 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 06:29:37.425158 12978 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 06:29:37.427320 12978 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 06:29:37.427454 12978 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 06:29:37.427685 12978 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 06:29:37.427973 12978 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 06:29:37.428004 12978 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 06:29:37.428025 12978 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 06:29:37.428046 12978 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 06:29:37.433399 12978 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.130:37609
I20260806 06:29:37.433459 13182 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.130:37609 every 8 connection(s)
I20260806 06:29:37.433758 12978 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/minicluster-data/ts-1/data/info.pb
I20260806 06:29:37.436595 21554 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHhLhT/build/release/bin/kudu as pid 12978
I20260806 06:29:37.436843 21554 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHhLhT/build/release/bin/kudu
/tmp/dist-test-taskFHhLhT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.12.131:43851
--local_ip_for_outbound_sockets=127.21.12.131
--tserver_master_addrs=127.21.12.190:35555
--webserver_port=35959
--webserver_interface=127.21.12.131
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.131.keytab
--principal=kudu/127.21.12.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.148:42623
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 06:29:37 dist-test-slave-4tdj krb5kdc[11829](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997777, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.130@KRBTEST.COM for kudu/127.21.12.190@KRBTEST.COM
I20260806 06:29:37.442672 12976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:29:37.434431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.130:52683 (local address 127.21.12.190:35555)
0806 06:29:37.434490 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0806 06:29:37.434492 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:29:37.434761 (+ 269us) server_negotiation.cc:407] Connection header received
0806 06:29:37.435255 (+ 494us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:29:37.435256 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:29:37.435273 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:29:37.435326 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:29:37.436240 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:29:37.436786 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:29:37.437763 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:29:37.437982 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:29:37.438039 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:29:37.440185 (+ 2146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:29:37.440187 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:29:37.440187 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 06:29:37.440195 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:29:37.441412 (+ 1217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:29:37.441773 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:29:37.441775 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:29:37.441775 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:29:37.441827 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:29:37.442035 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:29:37.442036 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:29:37.442037 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:29:37.442142 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:29:37.442196 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0806 06:29:37.442482 (+ 286us) server_negotiation.cc:299] Negotiation successful
0806 06:29:37.442582 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260806 06:29:37.442806 13190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:29:37.434459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:35555 (local address 127.21.12.130:52683)
0806 06:29:37.434643 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0806 06:29:37.434679 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0806 06:29:37.435163 (+ 484us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:29:37.435394 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:29:37.435398 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:29:37.435959 (+ 561us) client_negotiation.cc:189] Negotiated authn=SASL
0806 06:29:37.436148 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:29:37.436153 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:29:37.436864 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:29:37.436866 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:29:37.437682 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:29:37.437685 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:29:37.438050 (+ 365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:29:37.438051 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:29:37.438184 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:29:37.438646 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 06:29:37.438656 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0806 06:29:37.440097 (+ 1441us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 06:29:37.441487 (+ 1390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:29:37.441494 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:29:37.441496 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:29:37.441707 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:29:37.441880 (+ 173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:29:37.441881 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:29:37.441882 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:29:37.441981 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:29:37.442196 (+ 215us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 06:29:37.442202 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 06:29:37.442365 (+ 163us) client_negotiation.cc:769] Sending connection context
0806 06:29:37.442456 (+ 91us) client_negotiation.cc:240] Negotiation successful
0806 06:29:37.442605 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":104,"threads_started":1}
I20260806 06:29:37.443243 13183 heartbeater.cc:347] Connected to a master server at 127.21.12.190:35555
I20260806 06:29:37.443351 13183 heartbeater.cc:464] Registering TS with master...
I20260806 06:29:37.443552 13183 heartbeater.cc:511] Master 127.21.12.190:35555 requested a full tablet report, sending...
I20260806 06:29:37.443924 12680 ts_manager.cc:194] Registered new tserver with Master: 8c943bdb7a704767808dc5e9d578bc62 (127.21.12.130:37609)
I20260806 06:29:37.444399 12680 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.12.130@KRBTEST.COM'} at 127.21.12.130:52683
W20260806 06:29:37.518493 13191 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 06:29:37.518680 13191 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 06:29:37.518702 13191 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 06:29:37.519784 13191 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 06:29:37.519817 13191 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 06:29:37.519840 13191 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.12.131
I20260806 06:29:37.521265 13191 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.148:42623
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.131.keytab
--principal=kudu/127.21.12.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.12.131:43851
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.12.131
--webserver_port=35959
--tserver_master_addrs=127.21.12.190:35555
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13191
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.12.131
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 05:11:02 UTC on ccycloud.quasar-hvzbbz.root.comops.site
build id 4569617
I20260806 06:29:37.521443 13191 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 06:29:37.521612 13191 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 06:29:37.526224 13197 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:29:37.526383 13191 server_base.cc:1034] running on GCE node
W20260806 06:29:37.526237 13200 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:29:37.526247 13198 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:29:37.526665 13191 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 06:29:37.526870 13191 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 06:29:37.528026 13191 hybrid_clock.cc:648] HybridClock initialized: now 1785997777528004 us; error 39 us; skew 500 ppm
Aug 06 06:29:37 dist-test-slave-4tdj krb5kdc[11829](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997777, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 06:29:37.530499 13191 init.cc:377] Logged in from keytab as kudu/127.21.12.131@KRBTEST.COM (short username kudu)
I20260806 06:29:37.530915 13191 webserver.cc:460] Webserver started at http://127.21.12.131:35959/ using document root <none> and password file <none>
I20260806 06:29:37.531134 13191 fs_manager.cc:359] Metadata directory not provided
I20260806 06:29:37.531198 13191 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 06:29:37.532544 13191 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260806 06:29:37.533083 13208 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 06:29:37.533236 13191 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260806 06:29:37.533306 13191 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/minicluster-data/ts-2/wal
uuid: "ad7b449cdaa54d07b6ff51bf58c98948"
format_stamp: "Formatted at 2026-08-06 06:29:16 on dist-test-slave-4tdj"
server_key: "8b402f2674abfa9a402d4e86c883fcd1"
server_key_iv: "974ce2c7f328631a37b3107550f5261d"
server_key_version: "encryptionkey@0"
I20260806 06:29:37.533666 13191 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 06:29:37.546092 13191 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 06:29:37.548004 13191 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 06:29:37.548130 13191 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 06:29:37.548323 13191 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 06:29:37.548601 13191 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 06:29:37.548635 13191 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 06:29:37.548660 13191 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 06:29:37.548691 13191 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 06:29:37.554026 13191 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.131:43851
I20260806 06:29:37.554086 13382 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.131:43851 every 8 connection(s)
I20260806 06:29:37.554399 13191 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/minicluster-data/ts-2/data/info.pb
Aug 06 06:29:37 dist-test-slave-4tdj krb5kdc[11829](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997777, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.131@KRBTEST.COM for kudu/127.21.12.190@KRBTEST.COM
I20260806 06:29:37.562484 12976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:29:37.555057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.131:43127 (local address 127.21.12.190:35555)
0806 06:29:37.555114 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0806 06:29:37.555116 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:29:37.555324 (+ 208us) server_negotiation.cc:407] Connection header received
0806 06:29:37.555763 (+ 439us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:29:37.555764 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:29:37.555775 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:29:37.555833 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:29:37.556659 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:29:37.557126 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:29:37.558025 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:29:37.558216 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:29:37.558269 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:29:37.560194 (+ 1925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:29:37.560196 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:29:37.560196 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 06:29:37.560203 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:29:37.561430 (+ 1227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:29:37.561753 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:29:37.561754 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:29:37.561755 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:29:37.561814 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:29:37.561994 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:29:37.561994 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:29:37.561994 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:29:37.562087 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:29:37.562131 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0806 06:29:37.562339 (+ 208us) server_negotiation.cc:299] Negotiation successful
0806 06:29:37.562414 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260806 06:29:37.562613 13388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:29:37.555027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:35555 (local address 127.21.12.131:43127)
0806 06:29:37.555182 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0806 06:29:37.555221 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0806 06:29:37.555665 (+ 444us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:29:37.555897 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:29:37.555900 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:29:37.556416 (+ 516us) client_negotiation.cc:189] Negotiated authn=SASL
0806 06:29:37.556595 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:29:37.556599 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:29:37.557181 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:29:37.557182 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:29:37.557950 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:29:37.557952 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:29:37.558275 (+ 323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:29:37.558275 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:29:37.558410 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:29:37.558787 (+ 377us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 06:29:37.558795 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0806 06:29:37.560108 (+ 1313us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 06:29:37.561492 (+ 1384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:29:37.561497 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:29:37.561499 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:29:37.561691 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:29:37.561860 (+ 169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:29:37.561861 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:29:37.561861 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:29:37.561950 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:29:37.562127 (+ 177us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 06:29:37.562130 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 06:29:37.562226 (+ 96us) client_negotiation.cc:769] Sending connection context
0806 06:29:37.562323 (+ 97us) client_negotiation.cc:240] Negotiation successful
0806 06:29:37.562439 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":85,"threads_started":1}
I20260806 06:29:37.562858 21554 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHhLhT/build/release/bin/kudu as pid 13191
I20260806 06:29:37.563113 13383 heartbeater.cc:347] Connected to a master server at 127.21.12.190:35555
I20260806 06:29:37.563212 13383 heartbeater.cc:464] Registering TS with master...
I20260806 06:29:37.563395 13383 heartbeater.cc:511] Master 127.21.12.190:35555 requested a full tablet report, sending...
I20260806 06:29:37.563766 12680 ts_manager.cc:194] Registered new tserver with Master: ad7b449cdaa54d07b6ff51bf58c98948 (127.21.12.131:43851)
I20260806 06:29:37.564239 12680 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.12.131@KRBTEST.COM'} at 127.21.12.131:43127
I20260806 06:29:37.568475 12976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:29:37.563229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52000 (local address 127.21.12.190:35555)
0806 06:29:37.563288 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0806 06:29:37.563290 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:29:37.563444 (+ 154us) server_negotiation.cc:407] Connection header received
0806 06:29:37.563525 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:29:37.563526 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:29:37.563535 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:29:37.563580 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:29:37.564223 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:29:37.564718 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:29:37.565472 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:29:37.565641 (+ 169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:29:37.565681 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:29:37.566764 (+ 1083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:29:37.566765 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:29:37.566765 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 06:29:37.566770 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:29:37.567735 (+ 965us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:29:37.567998 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:29:37.567998 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:29:37.567999 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:29:37.568046 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:29:37.568154 (+ 108us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:29:37.568155 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:29:37.568155 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:29:37.568227 (+ 72us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:29:37.568337 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0806 06:29:37.568361 (+ 24us) server_negotiation.cc:299] Negotiation successful
0806 06:29:37.568412 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260806 06:29:37.569124 21554 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 06:29:37.571501 12976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:29:37.569409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52006 (local address 127.21.12.190:35555)
0806 06:29:37.569441 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0806 06:29:37.569443 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:29:37.569560 (+ 117us) server_negotiation.cc:407] Connection header received
0806 06:29:37.569605 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:29:37.569606 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:29:37.569632 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:29:37.569683 (+ 51us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 06:29:37.569756 (+ 73us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:29:37.570168 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:29:37.570925 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:29:37.571099 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:29:37.571144 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:29:37.571316 (+ 172us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 06:29:37.571353 (+ 37us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0806 06:29:37.571448 (+ 95us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:52006: token expired
Metrics: {"server-negotiator.queue_time_us":15}
W20260806 06:29:37.571542 12976 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:52006: token expired
W20260806 06:29:37.571756 12564 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.21.12.190:35555)
I20260806 06:29:37.576879 12976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:29:37.571990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52012 (local address 127.21.12.190:35555)
0806 06:29:37.572020 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0806 06:29:37.572022 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:29:37.572039 (+ 17us) server_negotiation.cc:407] Connection header received
0806 06:29:37.572073 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:29:37.572073 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:29:37.572080 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:29:37.572113 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:29:37.572488 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:29:37.572898 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:29:37.573516 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:29:37.573680 (+ 164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:29:37.573726 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:29:37.574664 (+ 938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:29:37.574665 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:29:37.574665 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 06:29:37.574671 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:29:37.575846 (+ 1175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:29:37.576175 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:29:37.576176 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:29:37.576177 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:29:37.576226 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:29:37.576452 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:29:37.576453 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:29:37.576453 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:29:37.576586 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:29:37.576667 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0806 06:29:37.576767 (+ 100us) server_negotiation.cc:299] Negotiation successful
0806 06:29:37.576825 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260806 06:29:37.577972 12680 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52012:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 06:29:37.579617 12680 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 06:29:37.581615 12877 tablet_service.cc:1467] Processing CreateTablet for tablet a13fddea3f8248858274754f1a5f7f42 (DEFAULT_TABLE table=test-table [id=b0fe066904e64aaca1390967acf37eeb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 06:29:37.581815 12877 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a13fddea3f8248858274754f1a5f7f42. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 06:29:37.583223 12810 tablet_bootstrap.cc:492] T a13fddea3f8248858274754f1a5f7f42 P e9deeb824da74ee98f7e811e0c61cfab: Bootstrap starting.
I20260806 06:29:37.583643 12810 tablet_bootstrap.cc:654] T a13fddea3f8248858274754f1a5f7f42 P e9deeb824da74ee98f7e811e0c61cfab: Neither blocks nor log segments found. Creating new log.
I20260806 06:29:37.585078 12982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:29:37.581356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.130:37609 (local address 127.0.0.1:42882)
0806 06:29:37.581401 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0806 06:29:37.581410 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0806 06:29:37.581444 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:29:37.581832 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:29:37.581833 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:29:37.581835 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 06:29:37.581876 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:29:37.581877 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:29:37.582864 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:29:37.582866 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:29:37.583837 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:29:37.583840 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:29:37.584879 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:29:37.584880 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:29:37.584936 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:29:37.584953 (+ 17us) client_negotiation.cc:769] Sending connection context
0806 06:29:37.584999 (+ 46us) client_negotiation.cc:240] Negotiation successful
0806 06:29:37.585034 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260806 06:29:37.585150 13392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:29:37.581445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42882 (local address 127.21.12.130:37609)
0806 06:29:37.581629 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0806 06:29:37.581634 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 06:29:37.581644 (+ 10us) server_negotiation.cc:407] Connection header received
0806 06:29:37.581685 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:29:37.581687 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:29:37.581743 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:29:37.581823 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 06:29:37.581936 (+ 113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:29:37.582799 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:29:37.583939 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:29:37.584796 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:29:37.584869 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:29:37.584921 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0806 06:29:37.585030 (+ 109us) server_negotiation.cc:299] Negotiation successful
0806 06:29:37.585068 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":96,"threads_started":1}
I20260806 06:29:37.585585 12810 tablet_bootstrap.cc:492] T a13fddea3f8248858274754f1a5f7f42 P e9deeb824da74ee98f7e811e0c61cfab: No bootstrap required, opened a new log
I20260806 06:29:37.585640 12810 ts_tablet_manager.cc:1397] T a13fddea3f8248858274754f1a5f7f42 P e9deeb824da74ee98f7e811e0c61cfab: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20260806 06:29:37.585794 12810 raft_consensus.cc:348] T a13fddea3f8248858274754f1a5f7f42 P e9deeb824da74ee98f7e811e0c61cfab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c943bdb7a704767808dc5e9d578bc62" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 37609 } } peers { permanent_uuid: "ad7b449cdaa54d07b6ff51bf58c98948" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 43851 } } peers { permanent_uuid: "e9deeb824da74ee98f7e811e0c61cfab" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 42233 } }
I20260806 06:29:37.585865 12810 raft_consensus.cc:374] T a13fddea3f8248858274754f1a5f7f42 P e9deeb824da74ee98f7e811e0c61cfab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 06:29:37.585915 12810 raft_consensus.cc:729] T a13fddea3f8248858274754f1a5f7f42 P e9deeb824da74ee98f7e811e0c61cfab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9deeb824da74ee98f7e811e0c61cfab, State: Initialized, Role: FOLLOWER
I20260806 06:29:37.585963 12810 consensus_queue.cc:260] T a13fddea3f8248858274754f1a5f7f42 P e9deeb824da74ee98f7e811e0c61cfab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c943bdb7a704767808dc5e9d578bc62" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 37609 } } peers { permanent_uuid: "ad7b449cdaa54d07b6ff51bf58c98948" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 43851 } } peers { permanent_uuid: "e9deeb824da74ee98f7e811e0c61cfab" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 42233 } }
I20260806 06:29:37.586073 12810 ts_tablet_manager.cc:1428] T a13fddea3f8248858274754f1a5f7f42 P e9deeb824da74ee98f7e811e0c61cfab: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260806 06:29:37.585970 13082 tablet_service.cc:1467] Processing CreateTablet for tablet a13fddea3f8248858274754f1a5f7f42 (DEFAULT_TABLE table=test-table [id=b0fe066904e64aaca1390967acf37eeb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 06:29:37.586334 13082 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a13fddea3f8248858274754f1a5f7f42. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 06:29:37.586637 13391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:29:37.581356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.131:43851 (local address 127.0.0.1:52362)
0806 06:29:37.583237 (+ 1881us) negotiation.cc:107] Waiting for socket to connect
0806 06:29:37.583248 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0806 06:29:37.583304 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:29:37.583494 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:29:37.583495 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:29:37.583497 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 06:29:37.583539 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:29:37.583540 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:29:37.584392 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:29:37.584393 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:29:37.585378 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:29:37.585381 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:29:37.586389 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:29:37.586391 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:29:37.586475 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:29:37.586479 (+ 4us) client_negotiation.cc:769] Sending connection context
0806 06:29:37.586528 (+ 49us) client_negotiation.cc:240] Negotiation successful
0806 06:29:37.586570 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1866,"mutex_wait_us":11,"thread_start_us":62,"threads_started":1}
I20260806 06:29:37.586956 13393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:29:37.581552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52362 (local address 127.21.12.131:43851)
0806 06:29:37.581720 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0806 06:29:37.581725 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 06:29:37.583359 (+ 1634us) server_negotiation.cc:407] Connection header received
0806 06:29:37.583399 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:29:37.583401 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:29:37.583416 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:29:37.583481 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 06:29:37.583588 (+ 107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:29:37.584320 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:29:37.585453 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:29:37.586270 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:29:37.586383 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:29:37.586438 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0806 06:29:37.586836 (+ 398us) server_negotiation.cc:299] Negotiation successful
0806 06:29:37.586877 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":98,"threads_started":1}
I20260806 06:29:37.587528 13279 tablet_service.cc:1467] Processing CreateTablet for tablet a13fddea3f8248858274754f1a5f7f42 (DEFAULT_TABLE table=test-table [id=b0fe066904e64aaca1390967acf37eeb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 06:29:37.587807 13279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a13fddea3f8248858274754f1a5f7f42. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 06:29:37.589295 13395 tablet_bootstrap.cc:492] T a13fddea3f8248858274754f1a5f7f42 P 8c943bdb7a704767808dc5e9d578bc62: Bootstrap starting.
I20260806 06:29:37.589948 13395 tablet_bootstrap.cc:654] T a13fddea3f8248858274754f1a5f7f42 P 8c943bdb7a704767808dc5e9d578bc62: Neither blocks nor log segments found. Creating new log.
I20260806 06:29:37.590556 13395 log.cc:824] T a13fddea3f8248858274754f1a5f7f42 P 8c943bdb7a704767808dc5e9d578bc62: Log is configured to *not* fsync() on all Append() calls
I20260806 06:29:37.590906 13400 tablet_bootstrap.cc:492] T a13fddea3f8248858274754f1a5f7f42 P ad7b449cdaa54d07b6ff51bf58c98948: Bootstrap starting.
I20260806 06:29:37.591308 13395 tablet_bootstrap.cc:492] T a13fddea3f8248858274754f1a5f7f42 P 8c943bdb7a704767808dc5e9d578bc62: No bootstrap required, opened a new log
I20260806 06:29:37.591364 13395 ts_tablet_manager.cc:1397] T a13fddea3f8248858274754f1a5f7f42 P 8c943bdb7a704767808dc5e9d578bc62: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260806 06:29:37.591472 13400 tablet_bootstrap.cc:654] T a13fddea3f8248858274754f1a5f7f42 P ad7b449cdaa54d07b6ff51bf58c98948: Neither blocks nor log segments found. Creating new log.
I20260806 06:29:37.591907 13400 log.cc:824] T a13fddea3f8248858274754f1a5f7f42 P ad7b449cdaa54d07b6ff51bf58c98948: Log is configured to *not* fsync() on all Append() calls
I20260806 06:29:37.592569 13400 tablet_bootstrap.cc:492] T a13fddea3f8248858274754f1a5f7f42 P ad7b449cdaa54d07b6ff51bf58c98948: No bootstrap required, opened a new log
I20260806 06:29:37.592629 13400 ts_tablet_manager.cc:1397] T a13fddea3f8248858274754f1a5f7f42 P ad7b449cdaa54d07b6ff51bf58c98948: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260806 06:29:37.593078 13395 raft_consensus.cc:348] T a13fddea3f8248858274754f1a5f7f42 P 8c943bdb7a704767808dc5e9d578bc62 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c943bdb7a704767808dc5e9d578bc62" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 37609 } } peers { permanent_uuid: "ad7b449cdaa54d07b6ff51bf58c98948" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 43851 } } peers { permanent_uuid: "e9deeb824da74ee98f7e811e0c61cfab" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 42233 } }
I20260806 06:29:37.593255 13395 raft_consensus.cc:374] T a13fddea3f8248858274754f1a5f7f42 P 8c943bdb7a704767808dc5e9d578bc62 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 06:29:37.593284 13395 raft_consensus.cc:729] T a13fddea3f8248858274754f1a5f7f42 P 8c943bdb7a704767808dc5e9d578bc62 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c943bdb7a704767808dc5e9d578bc62, State: Initialized, Role: FOLLOWER
I20260806 06:29:37.593395 13395 consensus_queue.cc:260] T a13fddea3f8248858274754f1a5f7f42 P 8c943bdb7a704767808dc5e9d578bc62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c943bdb7a704767808dc5e9d578bc62" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 37609 } } peers { permanent_uuid: "ad7b449cdaa54d07b6ff51bf58c98948" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 43851 } } peers { permanent_uuid: "e9deeb824da74ee98f7e811e0c61cfab" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 42233 } }
I20260806 06:29:37.593662 13395 ts_tablet_manager.cc:1428] T a13fddea3f8248858274754f1a5f7f42 P 8c943bdb7a704767808dc5e9d578bc62: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260806 06:29:37.593681 13183 heartbeater.cc:503] Master 127.21.12.190:35555 was elected leader, sending a full tablet report...
I20260806 06:29:37.594483 13400 raft_consensus.cc:348] T a13fddea3f8248858274754f1a5f7f42 P ad7b449cdaa54d07b6ff51bf58c98948 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c943bdb7a704767808dc5e9d578bc62" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 37609 } } peers { permanent_uuid: "ad7b449cdaa54d07b6ff51bf58c98948" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 43851 } } peers { permanent_uuid: "e9deeb824da74ee98f7e811e0c61cfab" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 42233 } }
I20260806 06:29:37.594645 13400 raft_consensus.cc:374] T a13fddea3f8248858274754f1a5f7f42 P ad7b449cdaa54d07b6ff51bf58c98948 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 06:29:37.594682 13400 raft_consensus.cc:729] T a13fddea3f8248858274754f1a5f7f42 P ad7b449cdaa54d07b6ff51bf58c98948 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad7b449cdaa54d07b6ff51bf58c98948, State: Initialized, Role: FOLLOWER
I20260806 06:29:37.594756 13400 consensus_queue.cc:260] T a13fddea3f8248858274754f1a5f7f42 P ad7b449cdaa54d07b6ff51bf58c98948 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c943bdb7a704767808dc5e9d578bc62" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 37609 } } peers { permanent_uuid: "ad7b449cdaa54d07b6ff51bf58c98948" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 43851 } } peers { permanent_uuid: "e9deeb824da74ee98f7e811e0c61cfab" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 42233 } }
I20260806 06:29:37.594945 13400 ts_tablet_manager.cc:1428] T a13fddea3f8248858274754f1a5f7f42 P ad7b449cdaa54d07b6ff51bf58c98948: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260806 06:29:37.594998 13383 heartbeater.cc:503] Master 127.21.12.190:35555 was elected leader, sending a full tablet report...
W20260806 06:29:37.684293 13184 tablet.cc:2369] T a13fddea3f8248858274754f1a5f7f42 P 8c943bdb7a704767808dc5e9d578bc62: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 06:29:37.697269 13406 raft_consensus.cc:482] T a13fddea3f8248858274754f1a5f7f42 P ad7b449cdaa54d07b6ff51bf58c98948 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 06:29:37.697397 13406 raft_consensus.cc:504] T a13fddea3f8248858274754f1a5f7f42 P ad7b449cdaa54d07b6ff51bf58c98948 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c943bdb7a704767808dc5e9d578bc62" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 37609 } } peers { permanent_uuid: "ad7b449cdaa54d07b6ff51bf58c98948" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 43851 } } peers { permanent_uuid: "e9deeb824da74ee98f7e811e0c61cfab" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 42233 } }
I20260806 06:29:37.697762 13406 leader_election.cc:288] T a13fddea3f8248858274754f1a5f7f42 P ad7b449cdaa54d07b6ff51bf58c98948 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8c943bdb7a704767808dc5e9d578bc62 (127.21.12.130:37609), e9deeb824da74ee98f7e811e0c61cfab (127.21.12.129:42233)
I20260806 06:29:37.701462 13392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:29:37.698043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.131:50347 (local address 127.21.12.130:37609)
0806 06:29:37.698083 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0806 06:29:37.698085 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:29:37.698094 (+ 9us) server_negotiation.cc:407] Connection header received
0806 06:29:37.698141 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:29:37.698142 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:29:37.698149 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:29:37.698207 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 06:29:37.698374 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:29:37.699149 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:29:37.700554 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:29:37.701119 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:29:37.701197 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:29:37.701208 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0806 06:29:37.701351 (+ 143us) server_negotiation.cc:299] Negotiation successful
0806 06:29:37.701396 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260806 06:29:37.701493 12983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:29:37.698034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.131:52387 (local address 127.21.12.129:42233)
0806 06:29:37.698083 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0806 06:29:37.698085 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:29:37.698197 (+ 112us) server_negotiation.cc:407] Connection header received
0806 06:29:37.698240 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:29:37.698241 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:29:37.698267 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:29:37.698343 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 06:29:37.698461 (+ 118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:29:37.699188 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:29:37.700554 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:29:37.701185 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:29:37.701247 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:29:37.701310 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0806 06:29:37.701381 (+ 71us) server_negotiation.cc:299] Negotiation successful
0806 06:29:37.701421 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260806 06:29:37.701467 13388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:29:37.697944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.130:37609 (local address 127.21.12.131:50347)
0806 06:29:37.698028 (+ 84us) negotiation.cc:107] Waiting for socket to connect
0806 06:29:37.698040 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 06:29:37.698082 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:29:37.698218 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:29:37.698219 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:29:37.698221 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 06:29:37.698312 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:29:37.698314 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:29:37.699218 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:29:37.699218 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:29:37.700445 (+ 1227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:29:37.700448 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:29:37.701192 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:29:37.701192 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:29:37.701278 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:29:37.701282 (+ 4us) client_negotiation.cc:769] Sending connection context
0806 06:29:37.701336 (+ 54us) client_negotiation.cc:240] Negotiation successful
0806 06:29:37.701380 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66,"mutex_wait_us":8}
I20260806 06:29:37.701474 13407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:29:37.697954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.129:42233 (local address 127.21.12.131:52387)
0806 06:29:37.698131 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0806 06:29:37.698139 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0806 06:29:37.698188 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:29:37.698348 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:29:37.698349 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:29:37.698351 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 06:29:37.698405 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:29:37.698406 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:29:37.699251 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:29:37.699252 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:29:37.700445 (+ 1193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:29:37.700448 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:29:37.701253 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:29:37.701254 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:29:37.701321 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:29:37.701324 (+ 3us) client_negotiation.cc:769] Sending connection context
0806 06:29:37.701365 (+ 41us) client_negotiation.cc:240] Negotiation successful
0806 06:29:37.701390 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"mutex_wait_us":29,"thread_start_us":98,"threads_started":1}
I20260806 06:29:37.701838 13107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a13fddea3f8248858274754f1a5f7f42" candidate_uuid: "ad7b449cdaa54d07b6ff51bf58c98948" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c943bdb7a704767808dc5e9d578bc62" is_pre_election: true
I20260806 06:29:37.701987 13107 raft_consensus.cc:2454] T a13fddea3f8248858274754f1a5f7f42 P 8c943bdb7a704767808dc5e9d578bc62 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad7b449cdaa54d07b6ff51bf58c98948 in term 0.
I20260806 06:29:37.701992 12904 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a13fddea3f8248858274754f1a5f7f42" candidate_uuid: "ad7b449cdaa54d07b6ff51bf58c98948" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9deeb824da74ee98f7e811e0c61cfab" is_pre_election: true
I20260806 06:29:37.702149 12904 raft_consensus.cc:2454] T a13fddea3f8248858274754f1a5f7f42 P e9deeb824da74ee98f7e811e0c61cfab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad7b449cdaa54d07b6ff51bf58c98948 in term 0.
I20260806 06:29:37.702234 13210 leader_election.cc:302] T a13fddea3f8248858274754f1a5f7f42 P ad7b449cdaa54d07b6ff51bf58c98948 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c943bdb7a704767808dc5e9d578bc62, ad7b449cdaa54d07b6ff51bf58c98948; no voters:
I20260806 06:29:37.702381 13406 raft_consensus.cc:2781] T a13fddea3f8248858274754f1a5f7f42 P ad7b449cdaa54d07b6ff51bf58c98948 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 06:29:37.702446 13406 raft_consensus.cc:482] T a13fddea3f8248858274754f1a5f7f42 P ad7b449cdaa54d07b6ff51bf58c98948 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 06:29:37.702471 13406 raft_consensus.cc:3037] T a13fddea3f8248858274754f1a5f7f42 P ad7b449cdaa54d07b6ff51bf58c98948 [term 0 FOLLOWER]: Advancing to term 1
I20260806 06:29:37.703189 13406 raft_consensus.cc:504] T a13fddea3f8248858274754f1a5f7f42 P ad7b449cdaa54d07b6ff51bf58c98948 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c943bdb7a704767808dc5e9d578bc62" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 37609 } } peers { permanent_uuid: "ad7b449cdaa54d07b6ff51bf58c98948" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 43851 } } peers { permanent_uuid: "e9deeb824da74ee98f7e811e0c61cfab" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 42233 } }
I20260806 06:29:37.703315 13406 leader_election.cc:288] T a13fddea3f8248858274754f1a5f7f42 P ad7b449cdaa54d07b6ff51bf58c98948 [CANDIDATE]: Term 1 election: Requested vote from peers 8c943bdb7a704767808dc5e9d578bc62 (127.21.12.130:37609), e9deeb824da74ee98f7e811e0c61cfab (127.21.12.129:42233)
I20260806 06:29:37.703455 12904 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a13fddea3f8248858274754f1a5f7f42" candidate_uuid: "ad7b449cdaa54d07b6ff51bf58c98948" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9deeb824da74ee98f7e811e0c61cfab"
I20260806 06:29:37.703466 13107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a13fddea3f8248858274754f1a5f7f42" candidate_uuid: "ad7b449cdaa54d07b6ff51bf58c98948" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c943bdb7a704767808dc5e9d578bc62"
I20260806 06:29:37.703534 12904 raft_consensus.cc:3037] T a13fddea3f8248858274754f1a5f7f42 P e9deeb824da74ee98f7e811e0c61cfab [term 0 FOLLOWER]: Advancing to term 1
I20260806 06:29:37.703539 13107 raft_consensus.cc:3037] T a13fddea3f8248858274754f1a5f7f42 P 8c943bdb7a704767808dc5e9d578bc62 [term 0 FOLLOWER]: Advancing to term 1
I20260806 06:29:37.704233 12904 raft_consensus.cc:2454] T a13fddea3f8248858274754f1a5f7f42 P e9deeb824da74ee98f7e811e0c61cfab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad7b449cdaa54d07b6ff51bf58c98948 in term 1.
I20260806 06:29:37.704241 13107 raft_consensus.cc:2454] T a13fddea3f8248858274754f1a5f7f42 P 8c943bdb7a704767808dc5e9d578bc62 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad7b449cdaa54d07b6ff51bf58c98948 in term 1.
I20260806 06:29:37.704393 13214 leader_election.cc:302] T a13fddea3f8248858274754f1a5f7f42 P ad7b449cdaa54d07b6ff51bf58c98948 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ad7b449cdaa54d07b6ff51bf58c98948, e9deeb824da74ee98f7e811e0c61cfab; no voters:
I20260806 06:29:37.704499 13406 raft_consensus.cc:2781] T a13fddea3f8248858274754f1a5f7f42 P ad7b449cdaa54d07b6ff51bf58c98948 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 06:29:37.704666 13406 raft_consensus.cc:686] T a13fddea3f8248858274754f1a5f7f42 P ad7b449cdaa54d07b6ff51bf58c98948 [term 1 LEADER]: Becoming Leader. State: Replica: ad7b449cdaa54d07b6ff51bf58c98948, State: Running, Role: LEADER
I20260806 06:29:37.704756 13406 consensus_queue.cc:237] T a13fddea3f8248858274754f1a5f7f42 P ad7b449cdaa54d07b6ff51bf58c98948 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c943bdb7a704767808dc5e9d578bc62" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 37609 } } peers { permanent_uuid: "ad7b449cdaa54d07b6ff51bf58c98948" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 43851 } } peers { permanent_uuid: "e9deeb824da74ee98f7e811e0c61cfab" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 42233 } }
I20260806 06:29:37.705633 12679 catalog_manager.cc:5495] T a13fddea3f8248858274754f1a5f7f42 P ad7b449cdaa54d07b6ff51bf58c98948 reported cstate change: term changed from 0 to 1, leader changed from <none> to ad7b449cdaa54d07b6ff51bf58c98948 (127.21.12.131). New cstate: current_term: 1 leader_uuid: "ad7b449cdaa54d07b6ff51bf58c98948" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c943bdb7a704767808dc5e9d578bc62" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 37609 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ad7b449cdaa54d07b6ff51bf58c98948" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 43851 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e9deeb824da74ee98f7e811e0c61cfab" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 42233 } health_report { overall_health: UNKNOWN } } }
I20260806 06:29:37.728117 13393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:29:37.725832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52368 (local address 127.21.12.131:43851)
0806 06:29:37.725885 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0806 06:29:37.725887 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:29:37.725895 (+ 8us) server_negotiation.cc:407] Connection header received
0806 06:29:37.725950 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:29:37.725951 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:29:37.725970 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:29:37.726044 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 06:29:37.726162 (+ 118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:29:37.726726 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:29:37.727410 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:29:37.727623 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:29:37.727678 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:29:37.727812 (+ 134us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 06:29:37.727912 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 06:29:37.727991 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0806 06:29:37.728007 (+ 16us) server_negotiation.cc:299] Negotiation successful
0806 06:29:37.728027 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260806 06:29:37.729708 13107 raft_consensus.cc:1261] T a13fddea3f8248858274754f1a5f7f42 P 8c943bdb7a704767808dc5e9d578bc62 [term 1 FOLLOWER]: Refusing update from remote peer ad7b449cdaa54d07b6ff51bf58c98948: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 06:29:37.729709 12904 raft_consensus.cc:1261] T a13fddea3f8248858274754f1a5f7f42 P e9deeb824da74ee98f7e811e0c61cfab [term 1 FOLLOWER]: Refusing update from remote peer ad7b449cdaa54d07b6ff51bf58c98948: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 06:29:37.730041 13408 consensus_queue.cc:1035] T a13fddea3f8248858274754f1a5f7f42 P ad7b449cdaa54d07b6ff51bf58c98948 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c943bdb7a704767808dc5e9d578bc62" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 37609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 06:29:37.730159 13406 consensus_queue.cc:1035] T a13fddea3f8248858274754f1a5f7f42 P ad7b449cdaa54d07b6ff51bf58c98948 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9deeb824da74ee98f7e811e0c61cfab" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 42233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 06:29:37.731961 13413 mvcc.cc:205] Tried to move back new op lower bound from 7315446897578532864 to 7315446897479442432. Current Snapshot: MvccSnapshot[applied={T|T < 7315446897578532864}]
I20260806 06:29:37.732029 13414 mvcc.cc:205] Tried to move back new op lower bound from 7315446897578532864 to 7315446897479442432. Current Snapshot: MvccSnapshot[applied={T|T < 7315446897578532864}]
I20260806 06:29:37.734638 12680 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52012:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 06:29:37.734730 12680 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52012:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 06:29:37.735370 12680 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b4138ced16fa403fa2a8038460439b20: Sending DeleteTablet for 3 replicas of tablet a13fddea3f8248858274754f1a5f7f42
I20260806 06:29:37.735673 21554 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHhLhT/build/release/bin/kudu with pid 12743
I20260806 06:29:37.735668 13279 tablet_service.cc:1514] Processing DeleteTablet for tablet a13fddea3f8248858274754f1a5f7f42 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 06:29:37 UTC) from {username='kudu', principal='kudu/127.21.12.190@KRBTEST.COM'} at 127.0.0.1:52362
I20260806 06:29:37.735705 13082 tablet_service.cc:1514] Processing DeleteTablet for tablet a13fddea3f8248858274754f1a5f7f42 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 06:29:37 UTC) from {username='kudu', principal='kudu/127.21.12.190@KRBTEST.COM'} at 127.0.0.1:42882
I20260806 06:29:37.740285 13426 tablet_replica.cc:330] T a13fddea3f8248858274754f1a5f7f42 P ad7b449cdaa54d07b6ff51bf58c98948: stopping tablet replica
I20260806 06:29:37.740399 13426 raft_consensus.cc:2229] T a13fddea3f8248858274754f1a5f7f42 P ad7b449cdaa54d07b6ff51bf58c98948 [term 1 LEADER]: Raft consensus shutting down.
I20260806 06:29:37.740511 13426 raft_consensus.cc:2258] T a13fddea3f8248858274754f1a5f7f42 P ad7b449cdaa54d07b6ff51bf58c98948 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 06:29:37.741415 13426 ts_tablet_manager.cc:1905] T a13fddea3f8248858274754f1a5f7f42 P ad7b449cdaa54d07b6ff51bf58c98948: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 06:29:37.743180 13426 ts_tablet_manager.cc:1918] T a13fddea3f8248858274754f1a5f7f42 P ad7b449cdaa54d07b6ff51bf58c98948: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 06:29:37.743328 13426 log.cc:1196] T a13fddea3f8248858274754f1a5f7f42 P ad7b449cdaa54d07b6ff51bf58c98948: Deleting WAL directory at /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/minicluster-data/ts-2/wal/wals/a13fddea3f8248858274754f1a5f7f42
I20260806 06:29:37.743784 13426 ts_tablet_manager.cc:1939] T a13fddea3f8248858274754f1a5f7f42 P ad7b449cdaa54d07b6ff51bf58c98948: Deleting consensus metadata
I20260806 06:29:37.743904 13427 tablet_replica.cc:330] T a13fddea3f8248858274754f1a5f7f42 P 8c943bdb7a704767808dc5e9d578bc62: stopping tablet replica
I20260806 06:29:37.744028 13427 raft_consensus.cc:2229] T a13fddea3f8248858274754f1a5f7f42 P 8c943bdb7a704767808dc5e9d578bc62 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 06:29:37.744139 13427 raft_consensus.cc:2258] T a13fddea3f8248858274754f1a5f7f42 P 8c943bdb7a704767808dc5e9d578bc62 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 06:29:37.744458 12658 catalog_manager.cc:4841] TS ad7b449cdaa54d07b6ff51bf58c98948 (127.21.12.131:43851): tablet a13fddea3f8248858274754f1a5f7f42 (table test-table [id=b0fe066904e64aaca1390967acf37eeb]) successfully deleted
I20260806 06:29:37.745450 13427 ts_tablet_manager.cc:1905] T a13fddea3f8248858274754f1a5f7f42 P 8c943bdb7a704767808dc5e9d578bc62: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 06:29:37.746985 13427 ts_tablet_manager.cc:1918] T a13fddea3f8248858274754f1a5f7f42 P 8c943bdb7a704767808dc5e9d578bc62: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 06:29:37.747058 13427 log.cc:1196] T a13fddea3f8248858274754f1a5f7f42 P 8c943bdb7a704767808dc5e9d578bc62: Deleting WAL directory at /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/minicluster-data/ts-1/wal/wals/a13fddea3f8248858274754f1a5f7f42
I20260806 06:29:37.747316 13427 ts_tablet_manager.cc:1939] T a13fddea3f8248858274754f1a5f7f42 P 8c943bdb7a704767808dc5e9d578bc62: Deleting consensus metadata
I20260806 06:29:37.747725 12655 catalog_manager.cc:4841] TS 8c943bdb7a704767808dc5e9d578bc62 (127.21.12.130:37609): tablet a13fddea3f8248858274754f1a5f7f42 (table test-table [id=b0fe066904e64aaca1390967acf37eeb]) successfully deleted
W20260806 06:29:37.755638 12657 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.12.129:42233 (error 108)
I20260806 06:29:37.756052 21554 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHhLhT/build/release/bin/kudu with pid 12978
I20260806 06:29:37.756155 13391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:29:37.756017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.129:42233 (local address 127.0.0.1:44840)
0806 06:29:37.756077 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0806 06:29:37.756123 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.129:42233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260806 06:29:37.759342 12657 catalog_manager.cc:4584] TS e9deeb824da74ee98f7e811e0c61cfab (127.21.12.129:42233): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a13fddea3f8248858274754f1a5f7f42: Network error: Client connection negotiation failed: client connection to 127.21.12.129:42233: connect: Connection refused (error 111)
I20260806 06:29:37.762681 21554 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHhLhT/build/release/bin/kudu with pid 13191
I20260806 06:29:37.769345 21554 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHhLhT/build/release/bin/kudu with pid 12628
I20260806 06:29:57.778435 21554 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHhLhT/build/release/bin/kudu
/tmp/dist-test-taskFHhLhT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.12.190:35555
--webserver_interface=127.21.12.190
--webserver_port=41507
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.190.keytab
--principal=kudu/127.21.12.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.148:42623
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.12.190:35555
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 06:29:57.859957 13430 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 06:29:57.860157 13430 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 06:29:57.860176 13430 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 06:29:57.861342 13430 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260806 06:29:57.861383 13430 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260806 06:29:57.861395 13430 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260806 06:29:57.861405 13430 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 06:29:57.861416 13430 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 06:29:57.861428 13430 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 06:29:57.861438 13430 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 06:29:57.861454 13430 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 06:29:57.862952 13430 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.148:42623
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.21.12.190:35555
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.190.keytab
--principal=kudu/127.21.12.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.12.190:35555
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.12.190
--webserver_port=41507
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13430
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 05:11:02 UTC on ccycloud.quasar-hvzbbz.root.comops.site
build id 4569617
I20260806 06:29:57.863178 13430 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 06:29:57.863390 13430 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 06:29:57.868081 13444 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:29:57.868139 13440 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:29:57.868127 13438 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:29:57.868269 13430 server_base.cc:1034] running on GCE node
I20260806 06:29:57.868525 13430 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 06:29:57.868831 13430 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 06:29:57.870004 13430 hybrid_clock.cc:648] HybridClock initialized: now 1785997797869988 us; error 28 us; skew 500 ppm
Aug 06 06:29:57 dist-test-slave-4tdj krb5kdc[11829](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997797, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 06:29:57.876214 13430 init.cc:377] Logged in from keytab as kudu/127.21.12.190@KRBTEST.COM (short username kudu)
I20260806 06:29:57.876642 13430 webserver.cc:460] Webserver started at http://127.21.12.190:41507/ using document root <none> and password file <none>
I20260806 06:29:57.876840 13430 fs_manager.cc:359] Metadata directory not provided
I20260806 06:29:57.876888 13430 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 06:29:57.878165 13430 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260806 06:29:57.878859 13455 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 06:29:57.879004 13430 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260806 06:29:57.879103 13430 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/minicluster-data/master-0/data,/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/minicluster-data/master-0/wal
uuid: "b4138ced16fa403fa2a8038460439b20"
format_stamp: "Formatted at 2026-08-06 06:29:15 on dist-test-slave-4tdj"
server_key: "98b1cf0691cf2d63394abfbe6245ad78"
server_key_iv: "5ba1b7f3ade5efc32f2f199f78ee440c"
server_key_version: "encryptionkey@0"
I20260806 06:29:57.879431 13430 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 06:29:57.912830 13430 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 06:29:57.914718 13430 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 06:29:57.914857 13430 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 06:29:57.918627 13430 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.190:35555
I20260806 06:29:57.918663 13539 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.190:35555 every 8 connection(s)
I20260806 06:29:57.919023 13430 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/minicluster-data/master-0/data/info.pb
I20260806 06:29:57.920557 13540 sys_catalog.cc:263] Verifying existing consensus state
I20260806 06:29:57.921212 13540 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b4138ced16fa403fa2a8038460439b20: Bootstrap starting.
I20260806 06:29:57.924484 21554 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHhLhT/build/release/bin/kudu as pid 13430
I20260806 06:29:57.924759 21554 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHhLhT/build/release/bin/kudu
/tmp/dist-test-taskFHhLhT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.12.129:42233
--local_ip_for_outbound_sockets=127.21.12.129
--tserver_master_addrs=127.21.12.190:35555
--webserver_port=39581
--webserver_interface=127.21.12.129
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.129.keytab
--principal=kudu/127.21.12.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.148:42623
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 06:29:57.929584 13540 log.cc:824] T 00000000000000000000000000000000 P b4138ced16fa403fa2a8038460439b20: Log is configured to *not* fsync() on all Append() calls
I20260806 06:29:57.931720 13540 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b4138ced16fa403fa2a8038460439b20: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 06:29:57.931996 13540 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b4138ced16fa403fa2a8038460439b20: Bootstrap complete.
I20260806 06:29:57.934042 13540 raft_consensus.cc:348] T 00000000000000000000000000000000 P b4138ced16fa403fa2a8038460439b20 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4138ced16fa403fa2a8038460439b20" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 35555 } }
I20260806 06:29:57.934257 13540 raft_consensus.cc:729] T 00000000000000000000000000000000 P b4138ced16fa403fa2a8038460439b20 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4138ced16fa403fa2a8038460439b20, State: Initialized, Role: FOLLOWER
I20260806 06:29:57.934427 13540 consensus_queue.cc:260] T 00000000000000000000000000000000 P b4138ced16fa403fa2a8038460439b20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4138ced16fa403fa2a8038460439b20" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 35555 } }
I20260806 06:29:57.934489 13540 raft_consensus.cc:388] T 00000000000000000000000000000000 P b4138ced16fa403fa2a8038460439b20 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 06:29:57.934527 13540 raft_consensus.cc:482] T 00000000000000000000000000000000 P b4138ced16fa403fa2a8038460439b20 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 06:29:57.934581 13540 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b4138ced16fa403fa2a8038460439b20 [term 2 FOLLOWER]: Advancing to term 3
I20260806 06:29:57.935487 13540 raft_consensus.cc:504] T 00000000000000000000000000000000 P b4138ced16fa403fa2a8038460439b20 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4138ced16fa403fa2a8038460439b20" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 35555 } }
I20260806 06:29:57.935585 13540 leader_election.cc:302] T 00000000000000000000000000000000 P b4138ced16fa403fa2a8038460439b20 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b4138ced16fa403fa2a8038460439b20; no voters:
I20260806 06:29:57.935737 13540 leader_election.cc:288] T 00000000000000000000000000000000 P b4138ced16fa403fa2a8038460439b20 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260806 06:29:57.935781 13546 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b4138ced16fa403fa2a8038460439b20 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 06:29:57.935946 13546 raft_consensus.cc:686] T 00000000000000000000000000000000 P b4138ced16fa403fa2a8038460439b20 [term 3 LEADER]: Becoming Leader. State: Replica: b4138ced16fa403fa2a8038460439b20, State: Running, Role: LEADER
I20260806 06:29:57.936002 13540 sys_catalog.cc:564] T 00000000000000000000000000000000 P b4138ced16fa403fa2a8038460439b20 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 06:29:57.936036 13546 consensus_queue.cc:237] T 00000000000000000000000000000000 P b4138ced16fa403fa2a8038460439b20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4138ced16fa403fa2a8038460439b20" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 35555 } }
I20260806 06:29:57.936326 13546 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4138ced16fa403fa2a8038460439b20 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b4138ced16fa403fa2a8038460439b20. Latest consensus state: current_term: 3 leader_uuid: "b4138ced16fa403fa2a8038460439b20" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4138ced16fa403fa2a8038460439b20" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 35555 } } }
I20260806 06:29:57.936553 13546 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4138ced16fa403fa2a8038460439b20 [sys.catalog]: This master's current role is: LEADER
I20260806 06:29:57.936484 13547 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4138ced16fa403fa2a8038460439b20 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "b4138ced16fa403fa2a8038460439b20" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4138ced16fa403fa2a8038460439b20" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 35555 } } }
I20260806 06:29:57.936910 13547 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4138ced16fa403fa2a8038460439b20 [sys.catalog]: This master's current role is: LEADER
I20260806 06:29:57.937040 13556 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 06:29:57.937852 13556 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 06:29:57.938005 13556 catalog_manager.cc:1259] Loaded cluster ID: 46036382e0414cc0b117e7ffa1720aff
I20260806 06:29:57.938190 13556 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 06:29:57.939477 13556 catalog_manager.cc:1504] Loading token signing keys...
I20260806 06:29:57.939637 13556 catalog_manager.cc:5879] T 00000000000000000000000000000000 P b4138ced16fa403fa2a8038460439b20: Loaded TSK: 0
I20260806 06:29:57.939670 13556 catalog_manager.cc:5879] T 00000000000000000000000000000000 P b4138ced16fa403fa2a8038460439b20: Loaded TSK: 1
F20260806 06:29:57.939898 13556 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f94cf9619ad google::LogMessage::Fail() at ??:0
@ 0x7f94cf96488a google::LogMessage::SendToLog() at ??:0
@ 0x7f94cf9614de google::LogMessage::Flush() at ??:0
@ 0x7f94cf962a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f94d20ce104 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f94d20cbd41 kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f94d78bb1cd kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f94d78bb48e kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f94d78bb594 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f94d7881c48 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7f94d788bad0 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f94d182f0d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f94d1824954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f94d26886db start_thread at ??:0
@ 0x7f94ce45f61f clone at ??:0
W20260806 06:29:58.045326 13543 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 06:29:58.045753 13543 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 06:29:58.045806 13543 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 06:29:58.047565 13543 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 06:29:58.047626 13543 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 06:29:58.047663 13543 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.12.129
I20260806 06:29:58.049798 13543 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.148:42623
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.129.keytab
--principal=kudu/127.21.12.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.12.129:42233
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.12.129
--webserver_port=39581
--tserver_master_addrs=127.21.12.190:35555
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13543
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.12.129
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 05:11:02 UTC on ccycloud.quasar-hvzbbz.root.comops.site
build id 4569617
I20260806 06:29:58.050055 13543 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 06:29:58.050356 13543 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 06:29:58.055812 13598 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:29:58.055913 13543 server_base.cc:1034] running on GCE node
W20260806 06:29:58.055845 13602 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:29:58.055976 13599 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:29:58.056207 13543 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 06:29:58.056506 13543 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 06:29:58.057698 13543 hybrid_clock.cc:648] HybridClock initialized: now 1785997798057674 us; error 38 us; skew 500 ppm
Aug 06 06:29:58 dist-test-slave-4tdj krb5kdc[11829](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997798, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 06:29:58.060755 13543 init.cc:377] Logged in from keytab as kudu/127.21.12.129@KRBTEST.COM (short username kudu)
I20260806 06:29:58.061296 13543 webserver.cc:460] Webserver started at http://127.21.12.129:39581/ using document root <none> and password file <none>
I20260806 06:29:58.061594 13543 fs_manager.cc:359] Metadata directory not provided
I20260806 06:29:58.061664 13543 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 06:29:58.063413 13543 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260806 06:29:58.064182 13612 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 06:29:58.064344 13543 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260806 06:29:58.064442 13543 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/minicluster-data/ts-0/wal
uuid: "e9deeb824da74ee98f7e811e0c61cfab"
format_stamp: "Formatted at 2026-08-06 06:29:16 on dist-test-slave-4tdj"
server_key: "0f840dd02a07cb7da5a5a90045702fdf"
server_key_iv: "a1916e6fc4f675c3cccfa630297c5803"
server_key_version: "encryptionkey@0"
I20260806 06:29:58.064874 13543 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 06:29:58.093937 13543 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 06:29:58.096423 13543 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 06:29:58.096592 13543 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 06:29:58.096870 13543 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 06:29:58.097405 13624 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260806 06:29:58.098944 13543 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260806 06:29:58.099004 13543 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260806 06:29:58.099038 13543 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260806 06:29:58.099924 13543 ts_tablet_manager.cc:610] Registered 1 tablets
I20260806 06:29:58.099969 13543 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260806 06:29:58.100042 13624 tablet_bootstrap.cc:492] T a13fddea3f8248858274754f1a5f7f42 P e9deeb824da74ee98f7e811e0c61cfab: Bootstrap starting.
I20260806 06:29:58.106977 13543 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.129:42233
I20260806 06:29:58.107129 13794 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.129:42233 every 8 connection(s)
I20260806 06:29:58.107558 13543 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/minicluster-data/ts-0/data/info.pb
I20260806 06:29:58.110493 13624 log.cc:824] T a13fddea3f8248858274754f1a5f7f42 P e9deeb824da74ee98f7e811e0c61cfab: Log is configured to *not* fsync() on all Append() calls
I20260806 06:29:58.110780 21554 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHhLhT/build/release/bin/kudu as pid 13543
I20260806 06:29:58.111085 21554 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHhLhT/build/release/bin/kudu
/tmp/dist-test-taskFHhLhT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.12.130:37609
--local_ip_for_outbound_sockets=127.21.12.130
--tserver_master_addrs=127.21.12.190:35555
--webserver_port=42879
--webserver_interface=127.21.12.130
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.130.keytab
--principal=kudu/127.21.12.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.148:42623
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 06:29:58.113052 13624 tablet_bootstrap.cc:492] T a13fddea3f8248858274754f1a5f7f42 P e9deeb824da74ee98f7e811e0c61cfab: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 06:29:58.113299 13624 tablet_bootstrap.cc:492] T a13fddea3f8248858274754f1a5f7f42 P e9deeb824da74ee98f7e811e0c61cfab: Bootstrap complete.
I20260806 06:29:58.114027 13624 ts_tablet_manager.cc:1397] T a13fddea3f8248858274754f1a5f7f42 P e9deeb824da74ee98f7e811e0c61cfab: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.010s
I20260806 06:29:58.115036 13624 raft_consensus.cc:348] T a13fddea3f8248858274754f1a5f7f42 P e9deeb824da74ee98f7e811e0c61cfab [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c943bdb7a704767808dc5e9d578bc62" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 37609 } } peers { permanent_uuid: "ad7b449cdaa54d07b6ff51bf58c98948" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 43851 } } peers { permanent_uuid: "e9deeb824da74ee98f7e811e0c61cfab" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 42233 } }
I20260806 06:29:58.115259 13624 raft_consensus.cc:729] T a13fddea3f8248858274754f1a5f7f42 P e9deeb824da74ee98f7e811e0c61cfab [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9deeb824da74ee98f7e811e0c61cfab, State: Initialized, Role: FOLLOWER
I20260806 06:29:58.115433 13624 consensus_queue.cc:260] T a13fddea3f8248858274754f1a5f7f42 P e9deeb824da74ee98f7e811e0c61cfab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c943bdb7a704767808dc5e9d578bc62" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 37609 } } peers { permanent_uuid: "ad7b449cdaa54d07b6ff51bf58c98948" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 43851 } } peers { permanent_uuid: "e9deeb824da74ee98f7e811e0c61cfab" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 42233 } }
I20260806 06:29:58.115691 13624 ts_tablet_manager.cc:1428] T a13fddea3f8248858274754f1a5f7f42 P e9deeb824da74ee98f7e811e0c61cfab: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
W20260806 06:29:58.159860 13797 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 06:29:58.108262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:35555 (local address 127.21.12.129:54705)
0806 06:29:58.108427 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0806 06:29:58.108474 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0806 06:29:58.109121 (+ 647us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:29:58.159652 (+ 50531us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.190:35555: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":76,"threads_started":1}
W20260806 06:29:58.160117 13795 heartbeater.cc:644] Failed to heartbeat to 127.21.12.190:35555 (0 consecutive failures): Network error: Failed to ping master at 127.21.12.190:35555: Client connection negotiation failed: client connection to 127.21.12.190:35555: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260806 06:29:58.160538 13797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:29:58.160422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:35555 (local address 127.21.12.129:33661)
0806 06:29:58.160471 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0806 06:29:58.160513 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.190:35555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260806 06:29:58.160877 13797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:29:58.160794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:35555 (local address 127.21.12.129:35513)
0806 06:29:58.160836 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0806 06:29:58.160861 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.190:35555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260806 06:29:58.160986 13795 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260806 06:29:58.198768 13799 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 06:29:58.198963 13799 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 06:29:58.198984 13799 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 06:29:58.200058 13799 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 06:29:58.200089 13799 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 06:29:58.200114 13799 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.12.130
I20260806 06:29:58.201506 13799 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.148:42623
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.130.keytab
--principal=kudu/127.21.12.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.12.130:37609
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.12.130
--webserver_port=42879
--tserver_master_addrs=127.21.12.190:35555
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13799
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.12.130
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 05:11:02 UTC on ccycloud.quasar-hvzbbz.root.comops.site
build id 4569617
I20260806 06:29:58.201689 13799 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 06:29:58.201858 13799 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 06:29:58.206619 13809 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:29:58.206615 13814 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:29:58.206648 13799 server_base.cc:1034] running on GCE node
W20260806 06:29:58.206650 13811 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:29:58.206970 13799 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 06:29:58.207196 13799 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 06:29:58.208343 13799 hybrid_clock.cc:648] HybridClock initialized: now 1785997798208328 us; error 30 us; skew 500 ppm
Aug 06 06:29:58 dist-test-slave-4tdj krb5kdc[11829](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997798, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 06:29:58.210783 13799 init.cc:377] Logged in from keytab as kudu/127.21.12.130@KRBTEST.COM (short username kudu)
I20260806 06:29:58.211184 13799 webserver.cc:460] Webserver started at http://127.21.12.130:42879/ using document root <none> and password file <none>
I20260806 06:29:58.211387 13799 fs_manager.cc:359] Metadata directory not provided
I20260806 06:29:58.211436 13799 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 06:29:58.212756 13799 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260806 06:29:58.213269 13826 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 06:29:58.213438 13799 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260806 06:29:58.213547 13799 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/minicluster-data/ts-1/wal
uuid: "8c943bdb7a704767808dc5e9d578bc62"
format_stamp: "Formatted at 2026-08-06 06:29:16 on dist-test-slave-4tdj"
server_key: "94402ca88765c1be95e453978be9f69b"
server_key_iv: "86cdf8c29f29fd9a4765778ff5631673"
server_key_version: "encryptionkey@0"
I20260806 06:29:58.213879 13799 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 06:29:58.226828 13799 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 06:29:58.228632 13799 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 06:29:58.228772 13799 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 06:29:58.228982 13799 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 06:29:58.229274 13799 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 06:29:58.229306 13799 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 06:29:58.229336 13799 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 06:29:58.229351 13799 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 06:29:58.235005 13799 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.130:37609
I20260806 06:29:58.235071 14001 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.130:37609 every 8 connection(s)
I20260806 06:29:58.235374 13799 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/minicluster-data/ts-1/data/info.pb
I20260806 06:29:58.236340 14005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:29:58.235988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:35555 (local address 127.21.12.130:51855)
0806 06:29:58.236126 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0806 06:29:58.236200 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.190:35555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":70,"threads_started":1}
W20260806 06:29:58.236742 14002 heartbeater.cc:644] Failed to heartbeat to 127.21.12.190:35555 (0 consecutive failures): Network error: Failed to ping master at 127.21.12.190:35555: Client connection negotiation failed: client connection to 127.21.12.190:35555: connect: Connection refused (error 111)
I20260806 06:29:58.237001 14005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:29:58.236919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:35555 (local address 127.21.12.130:39601)
0806 06:29:58.236959 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0806 06:29:58.236984 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.190:35555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260806 06:29:58.237246 14005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:29:58.237173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:35555 (local address 127.21.12.130:34119)
0806 06:29:58.237207 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0806 06:29:58.237228 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.190:35555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
W20260806 06:29:58.237331 14002 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260806 06:29:58.238595 21554 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHhLhT/build/release/bin/kudu as pid 13799
I20260806 06:29:58.238843 21554 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHhLhT/build/release/bin/kudu
/tmp/dist-test-taskFHhLhT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.12.131:43851
--local_ip_for_outbound_sockets=127.21.12.131
--tserver_master_addrs=127.21.12.190:35555
--webserver_port=35959
--webserver_interface=127.21.12.131
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.131.keytab
--principal=kudu/127.21.12.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.148:42623
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 06:29:58.318428 14007 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 06:29:58.318624 14007 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 06:29:58.318655 14007 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 06:29:58.319784 14007 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 06:29:58.319823 14007 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 06:29:58.319857 14007 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.12.131
I20260806 06:29:58.321376 14007 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.148:42623
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.131.keytab
--principal=kudu/127.21.12.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.12.131:43851
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.12.131
--webserver_port=35959
--tserver_master_addrs=127.21.12.190:35555
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14007
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.12.131
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 05:11:02 UTC on ccycloud.quasar-hvzbbz.root.comops.site
build id 4569617
I20260806 06:29:58.321615 14007 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 06:29:58.321811 14007 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 06:29:58.326766 14015 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:29:58.326817 14017 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:29:58.326828 14014 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:29:58.326937 14007 server_base.cc:1034] running on GCE node
I20260806 06:29:58.327160 14007 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 06:29:58.327345 14007 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 06:29:58.328477 14007 hybrid_clock.cc:648] HybridClock initialized: now 1785997798328468 us; error 27 us; skew 500 ppm
Aug 06 06:29:58 dist-test-slave-4tdj krb5kdc[11829](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997798, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 06:29:58.330906 14007 init.cc:377] Logged in from keytab as kudu/127.21.12.131@KRBTEST.COM (short username kudu)
I20260806 06:29:58.331286 14007 webserver.cc:460] Webserver started at http://127.21.12.131:35959/ using document root <none> and password file <none>
I20260806 06:29:58.331454 14007 fs_manager.cc:359] Metadata directory not provided
I20260806 06:29:58.331492 14007 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 06:29:58.333084 14007 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260806 06:29:58.333633 14025 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 06:29:58.333761 14007 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260806 06:29:58.333829 14007 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/minicluster-data/ts-2/wal
uuid: "ad7b449cdaa54d07b6ff51bf58c98948"
format_stamp: "Formatted at 2026-08-06 06:29:16 on dist-test-slave-4tdj"
server_key: "8b402f2674abfa9a402d4e86c883fcd1"
server_key_iv: "974ce2c7f328631a37b3107550f5261d"
server_key_version: "encryptionkey@0"
I20260806 06:29:58.334147 14007 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 06:29:58.357070 14007 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 06:29:58.359200 14007 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 06:29:58.359309 14007 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 06:29:58.359496 14007 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 06:29:58.359745 14007 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 06:29:58.359773 14007 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 06:29:58.359800 14007 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 06:29:58.359813 14007 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 06:29:58.365188 14007 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.131:43851
I20260806 06:29:58.365258 14214 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.131:43851 every 8 connection(s)
I20260806 06:29:58.365576 14007 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785997581049705-21554-0/minicluster-data/ts-2/data/info.pb
I20260806 06:29:58.366645 14219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:29:58.366202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:35555 (local address 127.21.12.131:48279)
0806 06:29:58.366405 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0806 06:29:58.366480 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.190:35555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":100,"threads_started":1}
W20260806 06:29:58.367024 14216 heartbeater.cc:644] Failed to heartbeat to 127.21.12.190:35555 (0 consecutive failures): Network error: Failed to ping master at 127.21.12.190:35555: Client connection negotiation failed: client connection to 127.21.12.190:35555: connect: Connection refused (error 111)
I20260806 06:29:58.367266 14219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:29:58.367186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:35555 (local address 127.21.12.131:47595)
0806 06:29:58.367222 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0806 06:29:58.367249 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.190:35555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
I20260806 06:29:58.367463 14219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:29:58.367406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:35555 (local address 127.21.12.131:46809)
0806 06:29:58.367433 (+ 27us) negotiation.cc:107] Waiting for socket to connect
0806 06:29:58.367451 (+ 18us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.190:35555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
W20260806 06:29:58.367579 14216 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260806 06:29:58.374064 21554 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHhLhT/build/release/bin/kudu as pid 14007
I20260806 06:29:59.161746 14223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:29:59.161473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:35555 (local address 127.21.12.129:46311)
0806 06:29:59.161626 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0806 06:29:59.161677 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.190:35555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":70,"threads_started":1}
I20260806 06:29:59.237879 14226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:29:59.237679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:35555 (local address 127.21.12.130:49393)
0806 06:29:59.237792 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0806 06:29:59.237834 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.190:35555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":56,"threads_started":1}
I20260806 06:29:59.368139 14227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:29:59.367935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:35555 (local address 127.21.12.131:55717)
0806 06:29:59.368051 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0806 06:29:59.368092 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.190:35555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":65,"threads_started":1}
I20260806 06:29:59.556543 14228 raft_consensus.cc:482] T a13fddea3f8248858274754f1a5f7f42 P e9deeb824da74ee98f7e811e0c61cfab [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 06:29:59.556675 14228 raft_consensus.cc:504] T a13fddea3f8248858274754f1a5f7f42 P e9deeb824da74ee98f7e811e0c61cfab [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c943bdb7a704767808dc5e9d578bc62" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 37609 } } peers { permanent_uuid: "ad7b449cdaa54d07b6ff51bf58c98948" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 43851 } } peers { permanent_uuid: "e9deeb824da74ee98f7e811e0c61cfab" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 42233 } }
I20260806 06:29:59.557032 14228 leader_election.cc:288] T a13fddea3f8248858274754f1a5f7f42 P e9deeb824da74ee98f7e811e0c61cfab [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8c943bdb7a704767808dc5e9d578bc62 (127.21.12.130:37609), ad7b449cdaa54d07b6ff51bf58c98948 (127.21.12.131:43851)
Aug 06 06:29:59 dist-test-slave-4tdj krb5kdc[11829](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997798, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.129@KRBTEST.COM for kudu/127.21.12.131@KRBTEST.COM
Aug 06 06:29:59 dist-test-slave-4tdj krb5kdc[11829](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997798, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.129@KRBTEST.COM for kudu/127.21.12.130@KRBTEST.COM
I20260806 06:29:59.566888 14234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:29:59.557344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.129:55955 (local address 127.21.12.130:37609)
0806 06:29:59.557568 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0806 06:29:59.557576 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 06:29:59.557605 (+ 29us) server_negotiation.cc:407] Connection header received
0806 06:29:59.558010 (+ 405us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:29:59.558013 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:29:59.558054 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:29:59.558187 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:29:59.558938 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:29:59.559780 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:29:59.560769 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:29:59.561152 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:29:59.561248 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:29:59.563856 (+ 2608us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:29:59.563860 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:29:59.563860 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 06:29:59.563870 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:29:59.565431 (+ 1561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:29:59.565819 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:29:59.565822 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:29:59.565823 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:29:59.565912 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:29:59.566118 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:29:59.566119 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:29:59.566120 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:29:59.566393 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:29:59.566484 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0806 06:29:59.566667 (+ 183us) server_negotiation.cc:299] Negotiation successful
0806 06:29:59.566782 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":64,"threads_started":1}
I20260806 06:29:59.566891 14232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:29:59.557334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.129:60723 (local address 127.21.12.131:43851)
0806 06:29:59.557518 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0806 06:29:59.557529 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 06:29:59.557563 (+ 34us) server_negotiation.cc:407] Connection header received
0806 06:29:59.558010 (+ 447us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:29:59.558013 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:29:59.558063 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:29:59.558185 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:29:59.558941 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:29:59.559784 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:29:59.560773 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:29:59.561152 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:29:59.561248 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:29:59.563520 (+ 2272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:29:59.563523 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:29:59.563524 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 06:29:59.563534 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:29:59.565144 (+ 1610us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:29:59.565601 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:29:59.565604 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:29:59.565604 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:29:59.565706 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:29:59.565965 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:29:59.565966 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:29:59.565966 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:29:59.566249 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:29:59.566339 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0806 06:29:59.566657 (+ 318us) server_negotiation.cc:299] Negotiation successful
0806 06:29:59.566786 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":67,"threads_started":1}
I20260806 06:29:59.566891 14229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:29:59.557221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.131:43851 (local address 127.21.12.129:60723)
0806 06:29:59.557408 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0806 06:29:59.557418 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0806 06:29:59.557476 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:29:59.558249 (+ 773us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:29:59.558251 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:29:59.558634 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0806 06:29:59.558823 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:29:59.558829 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:29:59.559885 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:29:59.559887 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:29:59.560685 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:29:59.560688 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:29:59.561237 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:29:59.561238 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:29:59.561348 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:29:59.561928 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 06:29:59.561933 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0806 06:29:59.563421 (+ 1488us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 06:29:59.565245 (+ 1824us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:29:59.565252 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:29:59.565254 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:29:59.565523 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:29:59.565773 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:29:59.565774 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:29:59.565774 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:29:59.565900 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:29:59.566338 (+ 438us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 06:29:59.566344 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 06:29:59.566480 (+ 136us) client_negotiation.cc:769] Sending connection context
0806 06:29:59.566579 (+ 99us) client_negotiation.cc:240] Negotiation successful
0806 06:29:59.566761 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"mutex_wait_us":126,"thread_start_us":69,"threads_started":1}
I20260806 06:29:59.566957 14223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:29:59.557219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.130:37609 (local address 127.21.12.129:55955)
0806 06:29:59.557270 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0806 06:29:59.557282 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 06:29:59.557346 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:29:59.558248 (+ 902us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:29:59.558251 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:29:59.558632 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0806 06:29:59.558823 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:29:59.558829 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:29:59.559876 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:29:59.559877 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:29:59.560684 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:29:59.560686 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:29:59.561234 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:29:59.561236 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:29:59.561347 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:29:59.561888 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 06:29:59.561897 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0806 06:29:59.563769 (+ 1872us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 06:29:59.565520 (+ 1751us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:29:59.565521 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:29:59.565521 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:29:59.565750 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:29:59.565968 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:29:59.565968 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:29:59.565969 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:29:59.566040 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:29:59.566458 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 06:29:59.566459 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 06:29:59.566495 (+ 36us) client_negotiation.cc:769] Sending connection context
0806 06:29:59.566606 (+ 111us) client_negotiation.cc:240] Negotiation successful
0806 06:29:59.566830 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29,"mutex_wait_us":157}
W20260806 06:29:59.567795 13618 leader_election.cc:341] T a13fddea3f8248858274754f1a5f7f42 P e9deeb824da74ee98f7e811e0c61cfab [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ad7b449cdaa54d07b6ff51bf58c98948 (127.21.12.131:43851): Not found: Tablet not found: a13fddea3f8248858274754f1a5f7f42
W20260806 06:29:59.568382 13614 leader_election.cc:341] T a13fddea3f8248858274754f1a5f7f42 P e9deeb824da74ee98f7e811e0c61cfab [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8c943bdb7a704767808dc5e9d578bc62 (127.21.12.130:37609): Not found: Tablet not found: a13fddea3f8248858274754f1a5f7f42
I20260806 06:29:59.568434 13614 leader_election.cc:302] T a13fddea3f8248858274754f1a5f7f42 P e9deeb824da74ee98f7e811e0c61cfab [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e9deeb824da74ee98f7e811e0c61cfab; no voters: 8c943bdb7a704767808dc5e9d578bc62, ad7b449cdaa54d07b6ff51bf58c98948
I20260806 06:29:59.568532 14228 raft_consensus.cc:2726] T a13fddea3f8248858274754f1a5f7f42 P e9deeb824da74ee98f7e811e0c61cfab [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 06:30:00.162469 14235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:00.162214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:35555 (local address 127.21.12.129:52801)
0806 06:30:00.162381 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0806 06:30:00.162422 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.190:35555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":65,"threads_started":1}
I20260806 06:30:00.238515 14236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:00.238323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:35555 (local address 127.21.12.130:41843)
0806 06:30:00.238438 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0806 06:30:00.238471 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.190:35555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":62,"threads_started":1}
I20260806 06:30:00.368696 14237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:00.368514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:35555 (local address 127.21.12.131:46057)
0806 06:30:00.368625 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0806 06:30:00.368661 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.190:35555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":58,"threads_started":1}
I20260806 06:30:01.163152 14238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:01.162923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:35555 (local address 127.21.12.129:49127)
0806 06:30:01.163051 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0806 06:30:01.163083 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.190:35555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":58,"threads_started":1}
I20260806 06:30:01.239135 14239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:01.238908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:35555 (local address 127.21.12.130:45571)
0806 06:30:01.239053 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0806 06:30:01.239090 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.190:35555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":68,"threads_started":1}
I20260806 06:30:01.369288 14241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:01.369105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:35555 (local address 127.21.12.131:41493)
0806 06:30:01.369221 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0806 06:30:01.369253 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.190:35555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":63,"threads_started":1}
I20260806 06:30:01.455179 14242 raft_consensus.cc:482] T a13fddea3f8248858274754f1a5f7f42 P e9deeb824da74ee98f7e811e0c61cfab [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 06:30:01.455284 14242 raft_consensus.cc:504] T a13fddea3f8248858274754f1a5f7f42 P e9deeb824da74ee98f7e811e0c61cfab [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c943bdb7a704767808dc5e9d578bc62" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 37609 } } peers { permanent_uuid: "ad7b449cdaa54d07b6ff51bf58c98948" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 43851 } } peers { permanent_uuid: "e9deeb824da74ee98f7e811e0c61cfab" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 42233 } }
I20260806 06:30:01.455457 14242 leader_election.cc:288] T a13fddea3f8248858274754f1a5f7f42 P e9deeb824da74ee98f7e811e0c61cfab [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8c943bdb7a704767808dc5e9d578bc62 (127.21.12.130:37609), ad7b449cdaa54d07b6ff51bf58c98948 (127.21.12.131:43851)
W20260806 06:30:01.455862 13614 leader_election.cc:341] T a13fddea3f8248858274754f1a5f7f42 P e9deeb824da74ee98f7e811e0c61cfab [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8c943bdb7a704767808dc5e9d578bc62 (127.21.12.130:37609): Not found: Tablet not found: a13fddea3f8248858274754f1a5f7f42
W20260806 06:30:01.455956 13618 leader_election.cc:341] T a13fddea3f8248858274754f1a5f7f42 P e9deeb824da74ee98f7e811e0c61cfab [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ad7b449cdaa54d07b6ff51bf58c98948 (127.21.12.131:43851): Not found: Tablet not found: a13fddea3f8248858274754f1a5f7f42
I20260806 06:30:01.456029 13618 leader_election.cc:302] T a13fddea3f8248858274754f1a5f7f42 P e9deeb824da74ee98f7e811e0c61cfab [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e9deeb824da74ee98f7e811e0c61cfab; no voters: 8c943bdb7a704767808dc5e9d578bc62, ad7b449cdaa54d07b6ff51bf58c98948
I20260806 06:30:01.456106 14242 raft_consensus.cc:2726] T a13fddea3f8248858274754f1a5f7f42 P e9deeb824da74ee98f7e811e0c61cfab [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 06:30:02.163808 14244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:02.163590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:35555 (local address 127.21.12.129:36959)
0806 06:30:02.163719 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0806 06:30:02.163760 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.190:35555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":68,"threads_started":1}
I20260806 06:30:02.239687 14248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:02.239500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:35555 (local address 127.21.12.130:58411)
0806 06:30:02.239613 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0806 06:30:02.239648 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.190:35555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":57,"threads_started":1}
I20260806 06:30:02.369854 14249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:02.369662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:35555 (local address 127.21.12.131:39347)
0806 06:30:02.369777 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0806 06:30:02.369815 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.190:35555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":60,"threads_started":1}
I20260806 06:30:03.164506 14250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:03.164261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:35555 (local address 127.21.12.129:35291)
0806 06:30:03.164412 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0806 06:30:03.164454 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.190:35555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":61,"threads_started":1}
I20260806 06:30:03.240289 14252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:03.240086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:35555 (local address 127.21.12.130:38483)
0806 06:30:03.240212 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0806 06:30:03.240248 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.190:35555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":61,"threads_started":1}
I20260806 06:30:03.370460 14255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:03.370258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:35555 (local address 127.21.12.131:57113)
0806 06:30:03.370388 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0806 06:30:03.370424 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.190:35555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":73,"threads_started":1}
I20260806 06:30:03.930945 14256 raft_consensus.cc:482] T a13fddea3f8248858274754f1a5f7f42 P e9deeb824da74ee98f7e811e0c61cfab [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 06:30:03.931052 14256 raft_consensus.cc:504] T a13fddea3f8248858274754f1a5f7f42 P e9deeb824da74ee98f7e811e0c61cfab [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c943bdb7a704767808dc5e9d578bc62" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 37609 } } peers { permanent_uuid: "ad7b449cdaa54d07b6ff51bf58c98948" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 43851 } } peers { permanent_uuid: "e9deeb824da74ee98f7e811e0c61cfab" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 42233 } }
I20260806 06:30:03.931223 14256 leader_election.cc:288] T a13fddea3f8248858274754f1a5f7f42 P e9deeb824da74ee98f7e811e0c61cfab [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8c943bdb7a704767808dc5e9d578bc62 (127.21.12.130:37609), ad7b449cdaa54d07b6ff51bf58c98948 (127.21.12.131:43851)
W20260806 06:30:03.931627 13618 leader_election.cc:341] T a13fddea3f8248858274754f1a5f7f42 P e9deeb824da74ee98f7e811e0c61cfab [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ad7b449cdaa54d07b6ff51bf58c98948 (127.21.12.131:43851): Not found: Tablet not found: a13fddea3f8248858274754f1a5f7f42
W20260806 06:30:03.931710 13614 leader_election.cc:341] T a13fddea3f8248858274754f1a5f7f42 P e9deeb824da74ee98f7e811e0c61cfab [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8c943bdb7a704767808dc5e9d578bc62 (127.21.12.130:37609): Not found: Tablet not found: a13fddea3f8248858274754f1a5f7f42
I20260806 06:30:03.931752 13614 leader_election.cc:302] T a13fddea3f8248858274754f1a5f7f42 P e9deeb824da74ee98f7e811e0c61cfab [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e9deeb824da74ee98f7e811e0c61cfab; no voters: 8c943bdb7a704767808dc5e9d578bc62, ad7b449cdaa54d07b6ff51bf58c98948
I20260806 06:30:03.931821 14256 raft_consensus.cc:2726] T a13fddea3f8248858274754f1a5f7f42 P e9deeb824da74ee98f7e811e0c61cfab [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 06:30:04.165148 14258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:04.164937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:35555 (local address 127.21.12.129:53977)
0806 06:30:04.165061 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0806 06:30:04.165097 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.190:35555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":60,"threads_started":1}
I20260806 06:30:04.240965 14259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:04.240737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:35555 (local address 127.21.12.130:57873)
0806 06:30:04.240884 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0806 06:30:04.240922 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.190:35555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":78,"threads_started":1}
I20260806 06:30:04.371057 14260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:04.370828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:35555 (local address 127.21.12.131:46357)
0806 06:30:04.370971 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0806 06:30:04.371019 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.190:35555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":73,"threads_started":1}
I20260806 06:30:05.165843 14262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:05.165603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:35555 (local address 127.21.12.129:56807)
0806 06:30:05.165752 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0806 06:30:05.165792 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.190:35555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":73,"threads_started":1}
I20260806 06:30:05.241580 14263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:05.241368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:35555 (local address 127.21.12.130:59151)
0806 06:30:05.241504 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0806 06:30:05.241539 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.190:35555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":58,"threads_started":1}
I20260806 06:30:05.371717 14264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:05.371504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:35555 (local address 127.21.12.131:45721)
0806 06:30:05.371633 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0806 06:30:05.371671 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.190:35555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":65,"threads_started":1}
I20260806 06:30:06.166476 14265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:06.166269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:35555 (local address 127.21.12.129:53617)
0806 06:30:06.166391 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0806 06:30:06.166427 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.190:35555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":72,"threads_started":1}
I20260806 06:30:06.242204 14268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:06.241987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:35555 (local address 127.21.12.130:37235)
0806 06:30:06.242129 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0806 06:30:06.242164 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.190:35555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":81,"threads_started":1}
I20260806 06:30:06.372316 14271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:06.372108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:35555 (local address 127.21.12.131:37095)
0806 06:30:06.372241 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0806 06:30:06.372276 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.190:35555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":79,"threads_started":1}
I20260806 06:30:06.608026 14272 raft_consensus.cc:482] T a13fddea3f8248858274754f1a5f7f42 P e9deeb824da74ee98f7e811e0c61cfab [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 06:30:06.608143 14272 raft_consensus.cc:504] T a13fddea3f8248858274754f1a5f7f42 P e9deeb824da74ee98f7e811e0c61cfab [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c943bdb7a704767808dc5e9d578bc62" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 37609 } } peers { permanent_uuid: "ad7b449cdaa54d07b6ff51bf58c98948" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 43851 } } peers { permanent_uuid: "e9deeb824da74ee98f7e811e0c61cfab" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 42233 } }
I20260806 06:30:06.608314 14272 leader_election.cc:288] T a13fddea3f8248858274754f1a5f7f42 P e9deeb824da74ee98f7e811e0c61cfab [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8c943bdb7a704767808dc5e9d578bc62 (127.21.12.130:37609), ad7b449cdaa54d07b6ff51bf58c98948 (127.21.12.131:43851)
W20260806 06:30:06.608707 13618 leader_election.cc:341] T a13fddea3f8248858274754f1a5f7f42 P e9deeb824da74ee98f7e811e0c61cfab [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ad7b449cdaa54d07b6ff51bf58c98948 (127.21.12.131:43851): Not found: Tablet not found: a13fddea3f8248858274754f1a5f7f42
W20260806 06:30:06.608795 13614 leader_election.cc:341] T a13fddea3f8248858274754f1a5f7f42 P e9deeb824da74ee98f7e811e0c61cfab [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8c943bdb7a704767808dc5e9d578bc62 (127.21.12.130:37609): Not found: Tablet not found: a13fddea3f8248858274754f1a5f7f42
I20260806 06:30:06.608855 13614 leader_election.cc:302] T a13fddea3f8248858274754f1a5f7f42 P e9deeb824da74ee98f7e811e0c61cfab [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e9deeb824da74ee98f7e811e0c61cfab; no voters: 8c943bdb7a704767808dc5e9d578bc62, ad7b449cdaa54d07b6ff51bf58c98948
I20260806 06:30:06.608944 14272 raft_consensus.cc:2726] T a13fddea3f8248858274754f1a5f7f42 P e9deeb824da74ee98f7e811e0c61cfab [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 06:30:07.167074 14273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:07.166871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:35555 (local address 127.21.12.129:44769)
0806 06:30:07.166995 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0806 06:30:07.167027 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.190:35555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":68,"threads_started":1}
I20260806 06:30:07.242801 14274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:07.242611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:35555 (local address 127.21.12.130:45929)
0806 06:30:07.242726 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0806 06:30:07.242760 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.190:35555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":64,"threads_started":1}
I20260806 06:30:07.372893 14275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:07.372675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:35555 (local address 127.21.12.131:52409)
0806 06:30:07.372814 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0806 06:30:07.372853 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.190:35555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":59,"threads_started":1}
I20260806 06:30:08.167749 14276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:08.167525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:35555 (local address 127.21.12.129:55177)
0806 06:30:08.167662 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0806 06:30:08.167703 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.190:35555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":73,"threads_started":1}
I20260806 06:30:08.243383 14277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:08.243194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:35555 (local address 127.21.12.130:39667)
0806 06:30:08.243311 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0806 06:30:08.243343 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.190:35555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":66,"threads_started":1}
I20260806 06:30:08.373517 14278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:08.373325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:35555 (local address 127.21.12.131:36009)
0806 06:30:08.373443 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0806 06:30:08.373478 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.190:35555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":61,"threads_started":1}
I20260806 06:30:09.168346 14281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:09.168127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:35555 (local address 127.21.12.129:59115)
0806 06:30:09.168260 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0806 06:30:09.168297 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.190:35555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":60,"threads_started":1}
I20260806 06:30:09.243969 14282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:09.243783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:35555 (local address 127.21.12.130:56325)
0806 06:30:09.243892 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0806 06:30:09.243926 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.190:35555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":61,"threads_started":1}
I20260806 06:30:09.374150 14287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:09.373942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:35555 (local address 127.21.12.131:50699)
0806 06:30:09.374063 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0806 06:30:09.374104 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.190:35555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":65,"threads_started":1}
I20260806 06:30:10.169001 14288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:10.168759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:35555 (local address 127.21.12.129:53079)
0806 06:30:10.168903 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0806 06:30:10.168942 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.190:35555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":61,"threads_started":1}
I20260806 06:30:10.244653 14289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:10.244422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:35555 (local address 127.21.12.130:51317)
0806 06:30:10.244574 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0806 06:30:10.244610 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.190:35555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":84,"threads_started":1}
I20260806 06:30:10.374871 14290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:10.374647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:35555 (local address 127.21.12.131:44265)
0806 06:30:10.374789 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0806 06:30:10.374827 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.190:35555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":90,"threads_started":1}
I20260806 06:30:11.169631 14292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:11.169391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:35555 (local address 127.21.12.129:46555)
0806 06:30:11.169519 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0806 06:30:11.169560 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.190:35555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":83,"threads_started":1}
I20260806 06:30:11.245309 14293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:11.245065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:35555 (local address 127.21.12.130:35427)
0806 06:30:11.245195 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0806 06:30:11.245250 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.190:35555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":63,"threads_started":1}
I20260806 06:30:11.375548 14294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:11.375315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:35555 (local address 127.21.12.131:58165)
0806 06:30:11.375440 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0806 06:30:11.375485 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.190:35555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":71,"threads_started":1}
I20260806 06:30:11.802026 14295 raft_consensus.cc:482] T a13fddea3f8248858274754f1a5f7f42 P e9deeb824da74ee98f7e811e0c61cfab [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 06:30:11.802130 14295 raft_consensus.cc:504] T a13fddea3f8248858274754f1a5f7f42 P e9deeb824da74ee98f7e811e0c61cfab [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c943bdb7a704767808dc5e9d578bc62" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 37609 } } peers { permanent_uuid: "ad7b449cdaa54d07b6ff51bf58c98948" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 43851 } } peers { permanent_uuid: "e9deeb824da74ee98f7e811e0c61cfab" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 42233 } }
I20260806 06:30:11.802336 14295 leader_election.cc:288] T a13fddea3f8248858274754f1a5f7f42 P e9deeb824da74ee98f7e811e0c61cfab [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8c943bdb7a704767808dc5e9d578bc62 (127.21.12.130:37609), ad7b449cdaa54d07b6ff51bf58c98948 (127.21.12.131:43851)
W20260806 06:30:11.802666 13618 leader_election.cc:341] T a13fddea3f8248858274754f1a5f7f42 P e9deeb824da74ee98f7e811e0c61cfab [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ad7b449cdaa54d07b6ff51bf58c98948 (127.21.12.131:43851): Not found: Tablet not found: a13fddea3f8248858274754f1a5f7f42
W20260806 06:30:11.802742 13614 leader_election.cc:341] T a13fddea3f8248858274754f1a5f7f42 P e9deeb824da74ee98f7e811e0c61cfab [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8c943bdb7a704767808dc5e9d578bc62 (127.21.12.130:37609): Not found: Tablet not found: a13fddea3f8248858274754f1a5f7f42
I20260806 06:30:11.802804 13614 leader_election.cc:302] T a13fddea3f8248858274754f1a5f7f42 P e9deeb824da74ee98f7e811e0c61cfab [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e9deeb824da74ee98f7e811e0c61cfab; no voters: 8c943bdb7a704767808dc5e9d578bc62, ad7b449cdaa54d07b6ff51bf58c98948
I20260806 06:30:11.802874 14295 raft_consensus.cc:2726] T a13fddea3f8248858274754f1a5f7f42 P e9deeb824da74ee98f7e811e0c61cfab [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 06:30:12.170244 14296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:12.170043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:35555 (local address 127.21.12.129:46587)
0806 06:30:12.170158 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0806 06:30:12.170193 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.190:35555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":63,"threads_started":1}
I20260806 06:30:12.245882 14297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:12.245691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:35555 (local address 127.21.12.130:43289)
0806 06:30:12.245805 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0806 06:30:12.245841 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.190:35555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":62,"threads_started":1}
I20260806 06:30:12.376179 14298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:12.375943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:35555 (local address 127.21.12.131:48355)
0806 06:30:12.376091 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0806 06:30:12.376128 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.190:35555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":88,"threads_started":1}
I20260806 06:30:13.170970 14300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:13.170773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:35555 (local address 127.21.12.129:40787)
0806 06:30:13.170886 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0806 06:30:13.170923 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.190:35555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":60,"threads_started":1}
I20260806 06:30:13.246542 14302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:13.246358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:35555 (local address 127.21.12.130:60175)
0806 06:30:13.246467 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0806 06:30:13.246499 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.190:35555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":60,"threads_started":1}
I20260806 06:30:13.376825 14303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:13.376648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:35555 (local address 127.21.12.131:49597)
0806 06:30:13.376756 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0806 06:30:13.376788 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.190:35555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":59,"threads_started":1}
I20260806 06:30:13.381492 21554 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHhLhT/build/release/bin/kudu with pid 13543
I20260806 06:30:13.391172 21554 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHhLhT/build/release/bin/kudu with pid 13799
I20260806 06:30:13.399106 21554 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHhLhT/build/release/bin/kudu with pid 14007
2026-08-06T06:30:13Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (58813 ms)
[----------] 30 tests from SecurityITest (232359 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785997581049705-21554-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 06:30:13 dist-test-slave-4tdj krb5kdc[14309](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 06:30:13 dist-test-slave-4tdj krb5kdc[14309](info): set up 2 sockets
Aug 06 06:30:13 dist-test-slave-4tdj krb5kdc[14309](info): commencing operation
krb5kdc: starting...
W20260806 06:30:14.510988 21554 mini_kdc.cc:121] Time spent starting KDC: real 1.088s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785997581049705-21554-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785997581049705-21554-0/krb5kdc/test-user.keytab.
Aug 06 06:30:14 dist-test-slave-4tdj krb5kdc[14309](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997814, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T06:30:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T06:30:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.12.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.12.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.12.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.190.keytab.
Entry for principal kudu/127.21.12.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.190.keytab.
I20260806 06:30:14.624497 21554 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHhLhT/build/release/bin/kudu
/tmp/dist-test-taskFHhLhT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785997581049705-21554-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785997581049705-21554-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785997581049705-21554-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785997581049705-21554-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.12.190:39233
--webserver_interface=127.21.12.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.190.keytab
--principal=kudu/127.21.12.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.148:36879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.12.190:39233
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785997581049705-21554-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785997581049705-21554-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785997581049705-21554-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 06:30:14.703429 14330 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 06:30:14.703634 14330 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 06:30:14.703663 14330 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 06:30:14.704777 14330 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 06:30:14.704816 14330 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 06:30:14.704834 14330 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 06:30:14.704850 14330 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 06:30:14.704865 14330 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 06:30:14.706352 14330 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.148:36879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785997581049705-21554-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785997581049705-21554-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.12.190:39233
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.190.keytab
--principal=kudu/127.21.12.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.12.190:39233
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785997581049705-21554-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.12.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14330
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785997581049705-21554-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 05:11:02 UTC on ccycloud.quasar-hvzbbz.root.comops.site
build id 4569617
I20260806 06:30:14.706588 14330 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 06:30:14.706799 14330 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 06:30:14.711670 14340 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:30:14.711658 14336 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:30:14.711670 14343 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:30:14.711774 14330 server_base.cc:1034] running on GCE node
I20260806 06:30:14.712147 14330 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 06:30:14.712431 14330 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 06:30:14.713587 14330 hybrid_clock.cc:648] HybridClock initialized: now 1785997814713566 us; error 35 us; skew 500 ppm
Aug 06 06:30:14 dist-test-slave-4tdj krb5kdc[14309](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997814, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 06:30:14.715958 14330 init.cc:377] Logged in from keytab as kudu/127.21.12.190@KRBTEST.COM (short username kudu)
I20260806 06:30:14.716334 14330 webserver.cc:460] Webserver started at http://127.21.12.190:38417/ using document root <none> and password file <none>
I20260806 06:30:14.716534 14330 fs_manager.cc:359] Metadata directory not provided
I20260806 06:30:14.716574 14330 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 06:30:14.716667 14330 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 06:30:14.717631 14330 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785997581049705-21554-0/minicluster-data/master-0/data/instance:
uuid: "6f886f0db3be4ddfb6f3113cab9f2a79"
format_stamp: "Formatted at 2026-08-06 06:30:14 on dist-test-slave-4tdj"
server_key: "7d82566507025c955c3fbad5bc6ea22d"
server_key_iv: "2e94dfbe1d3e75eb1e91b1cc5715fc5f"
server_key_version: "encryptionkey@0"
I20260806 06:30:14.717924 14330 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785997581049705-21554-0/minicluster-data/master-0/wal/instance:
uuid: "6f886f0db3be4ddfb6f3113cab9f2a79"
format_stamp: "Formatted at 2026-08-06 06:30:14 on dist-test-slave-4tdj"
server_key: "7d82566507025c955c3fbad5bc6ea22d"
server_key_iv: "2e94dfbe1d3e75eb1e91b1cc5715fc5f"
server_key_version: "encryptionkey@0"
I20260806 06:30:14.719144 14330 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260806 06:30:14.719796 14352 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 06:30:14.719946 14330 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260806 06:30:14.720026 14330 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785997581049705-21554-0/minicluster-data/master-0/data,/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785997581049705-21554-0/minicluster-data/master-0/wal
uuid: "6f886f0db3be4ddfb6f3113cab9f2a79"
format_stamp: "Formatted at 2026-08-06 06:30:14 on dist-test-slave-4tdj"
server_key: "7d82566507025c955c3fbad5bc6ea22d"
server_key_iv: "2e94dfbe1d3e75eb1e91b1cc5715fc5f"
server_key_version: "encryptionkey@0"
I20260806 06:30:14.720079 14330 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785997581049705-21554-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785997581049705-21554-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785997581049705-21554-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 06:30:14.735810 14330 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 06:30:14.737772 14330 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 06:30:14.737880 14330 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 06:30:14.741550 14330 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.190:39233
I20260806 06:30:14.741595 14427 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.190:39233 every 8 connection(s)
I20260806 06:30:14.741888 14330 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785997581049705-21554-0/minicluster-data/master-0/data/info.pb
I20260806 06:30:14.742745 14428 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 06:30:14.744925 14428 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f886f0db3be4ddfb6f3113cab9f2a79: Bootstrap starting.
I20260806 06:30:14.745486 14428 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6f886f0db3be4ddfb6f3113cab9f2a79: Neither blocks nor log segments found. Creating new log.
I20260806 06:30:14.745744 14428 log.cc:824] T 00000000000000000000000000000000 P 6f886f0db3be4ddfb6f3113cab9f2a79: Log is configured to *not* fsync() on all Append() calls
I20260806 06:30:14.746440 14428 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f886f0db3be4ddfb6f3113cab9f2a79: No bootstrap required, opened a new log
I20260806 06:30:14.747959 14428 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6f886f0db3be4ddfb6f3113cab9f2a79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f886f0db3be4ddfb6f3113cab9f2a79" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 39233 } }
I20260806 06:30:14.748143 14428 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6f886f0db3be4ddfb6f3113cab9f2a79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 06:30:14.748190 14428 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6f886f0db3be4ddfb6f3113cab9f2a79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f886f0db3be4ddfb6f3113cab9f2a79, State: Initialized, Role: FOLLOWER
I20260806 06:30:14.748325 14428 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6f886f0db3be4ddfb6f3113cab9f2a79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f886f0db3be4ddfb6f3113cab9f2a79" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 39233 } }
I20260806 06:30:14.748389 14428 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6f886f0db3be4ddfb6f3113cab9f2a79 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 06:30:14.748418 14428 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6f886f0db3be4ddfb6f3113cab9f2a79 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 06:30:14.748464 14428 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6f886f0db3be4ddfb6f3113cab9f2a79 [term 0 FOLLOWER]: Advancing to term 1
I20260806 06:30:14.748765 21554 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHhLhT/build/release/bin/kudu as pid 14330
I20260806 06:30:14.748859 21554 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785997581049705-21554-0/minicluster-data/master-0/wal/instance
I20260806 06:30:14.749006 14428 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6f886f0db3be4ddfb6f3113cab9f2a79 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f886f0db3be4ddfb6f3113cab9f2a79" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 39233 } }
I20260806 06:30:14.749042 21554 external_mini_cluster.cc:1428] Setting key 57a87c4f2d2876bf761590ff96448807
I20260806 06:30:14.749109 14428 leader_election.cc:302] T 00000000000000000000000000000000 P 6f886f0db3be4ddfb6f3113cab9f2a79 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6f886f0db3be4ddfb6f3113cab9f2a79; no voters:
I20260806 06:30:14.749271 14428 leader_election.cc:288] T 00000000000000000000000000000000 P 6f886f0db3be4ddfb6f3113cab9f2a79 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 06:30:14.749291 14432 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6f886f0db3be4ddfb6f3113cab9f2a79 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 06:30:14.749402 14432 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6f886f0db3be4ddfb6f3113cab9f2a79 [term 1 LEADER]: Becoming Leader. State: Replica: 6f886f0db3be4ddfb6f3113cab9f2a79, State: Running, Role: LEADER
I20260806 06:30:14.749503 14432 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6f886f0db3be4ddfb6f3113cab9f2a79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f886f0db3be4ddfb6f3113cab9f2a79" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 39233 } }
I20260806 06:30:14.749540 14428 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6f886f0db3be4ddfb6f3113cab9f2a79 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 06:30:14.750047 14435 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f886f0db3be4ddfb6f3113cab9f2a79 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6f886f0db3be4ddfb6f3113cab9f2a79. Latest consensus state: current_term: 1 leader_uuid: "6f886f0db3be4ddfb6f3113cab9f2a79" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f886f0db3be4ddfb6f3113cab9f2a79" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 39233 } } }
I20260806 06:30:14.750352 14435 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f886f0db3be4ddfb6f3113cab9f2a79 [sys.catalog]: This master's current role is: LEADER
I20260806 06:30:14.750253 14433 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f886f0db3be4ddfb6f3113cab9f2a79 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6f886f0db3be4ddfb6f3113cab9f2a79" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f886f0db3be4ddfb6f3113cab9f2a79" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 39233 } } }
I20260806 06:30:14.750622 14433 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f886f0db3be4ddfb6f3113cab9f2a79 [sys.catalog]: This master's current role is: LEADER
I20260806 06:30:14.750625 14451 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 06:30:14.751250 14451 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 06:30:14.752748 14451 catalog_manager.cc:1347] Generated new cluster ID: 98a82f49a7df48a8aa370f49444eacbe
I20260806 06:30:14.752799 14451 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 06 06:30:14 dist-test-slave-4tdj krb5kdc[14309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997814, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.12.190@KRBTEST.COM
I20260806 06:30:14.760489 14440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:14.749490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49796 (local address 127.21.12.190:39233)
0806 06:30:14.749636 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0806 06:30:14.749646 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 06:30:14.750000 (+ 354us) server_negotiation.cc:407] Connection header received
0806 06:30:14.750138 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:30:14.750146 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:30:14.750203 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:30:14.750356 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:30:14.751272 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:14.752647 (+ 1375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:14.753695 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:14.754345 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:14.754454 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:30:14.757083 (+ 2629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:30:14.757088 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:30:14.757088 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 06:30:14.757100 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:30:14.758941 (+ 1841us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:30:14.759334 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:30:14.759336 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:30:14.759337 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:30:14.759451 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:30:14.759648 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:30:14.759648 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:30:14.759648 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:30:14.759913 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:30:14.760052 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0806 06:30:14.760163 (+ 111us) server_negotiation.cc:299] Negotiation successful
0806 06:30:14.760286 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96,"thread_start_us":63,"threads_started":1}
I20260806 06:30:14.764009 14451 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 06:30:14.764456 14451 catalog_manager.cc:1504] Loading token signing keys...
I20260806 06:30:14.766633 14451 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6f886f0db3be4ddfb6f3113cab9f2a79: Generated new TSK 0
I20260806 06:30:14.766789 14451 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.12.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.12.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.12.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.129.keytab.
Entry for principal kudu/127.21.12.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.129.keytab.
I20260806 06:30:14.794160 21554 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHhLhT/build/release/bin/kudu
/tmp/dist-test-taskFHhLhT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785997581049705-21554-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785997581049705-21554-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785997581049705-21554-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785997581049705-21554-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.12.129:0
--local_ip_for_outbound_sockets=127.21.12.129
--webserver_interface=127.21.12.129
--webserver_port=0
--tserver_master_addrs=127.21.12.190:39233
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.129.keytab
--principal=kudu/127.21.12.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.148:36879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785997581049705-21554-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785997581049705-21554-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785997581049705-21554-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 06:30:14.874763 14473 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 06:30:14.874949 14473 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 06:30:14.874971 14473 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 06:30:14.876099 14473 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 06:30:14.876142 14473 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 06:30:14.876169 14473 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.12.129
I20260806 06:30:14.877595 14473 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.148:36879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785997581049705-21554-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785997581049705-21554-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.129.keytab
--principal=kudu/127.21.12.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.12.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785997581049705-21554-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.12.129
--webserver_port=0
--tserver_master_addrs=127.21.12.190:39233
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14473
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.12.129
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785997581049705-21554-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 05:11:02 UTC on ccycloud.quasar-hvzbbz.root.comops.site
build id 4569617
I20260806 06:30:14.877810 14473 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 06:30:14.878011 14473 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 06:30:14.883029 14488 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:30:14.883168 14473 server_base.cc:1034] running on GCE node
W20260806 06:30:14.883056 14484 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:30:14.883039 14483 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:30:14.883460 14473 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 06:30:14.883677 14473 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 06:30:14.884841 14473 hybrid_clock.cc:648] HybridClock initialized: now 1785997814884826 us; error 29 us; skew 500 ppm
Aug 06 06:30:14 dist-test-slave-4tdj krb5kdc[14309](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997814, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 06:30:14.887477 14473 init.cc:377] Logged in from keytab as kudu/127.21.12.129@KRBTEST.COM (short username kudu)
I20260806 06:30:14.887920 14473 webserver.cc:460] Webserver started at http://127.21.12.129:43501/ using document root <none> and password file <none>
I20260806 06:30:14.888120 14473 fs_manager.cc:359] Metadata directory not provided
I20260806 06:30:14.888167 14473 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 06:30:14.888294 14473 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 06:30:14.889351 14473 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785997581049705-21554-0/minicluster-data/ts-0/data/instance:
uuid: "b267d2a459e34fefb8d9654d7b63ff45"
format_stamp: "Formatted at 2026-08-06 06:30:14 on dist-test-slave-4tdj"
server_key: "6cd4336d685e1c529e84210c5e128bb0"
server_key_iv: "1906c4e46e4090ad34a59a181b3c668b"
server_key_version: "encryptionkey@0"
I20260806 06:30:14.889685 14473 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785997581049705-21554-0/minicluster-data/ts-0/wal/instance:
uuid: "b267d2a459e34fefb8d9654d7b63ff45"
format_stamp: "Formatted at 2026-08-06 06:30:14 on dist-test-slave-4tdj"
server_key: "6cd4336d685e1c529e84210c5e128bb0"
server_key_iv: "1906c4e46e4090ad34a59a181b3c668b"
server_key_version: "encryptionkey@0"
I20260806 06:30:14.890995 14473 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260806 06:30:14.891670 14506 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 06:30:14.891811 14473 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260806 06:30:14.891906 14473 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785997581049705-21554-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785997581049705-21554-0/minicluster-data/ts-0/wal
uuid: "b267d2a459e34fefb8d9654d7b63ff45"
format_stamp: "Formatted at 2026-08-06 06:30:14 on dist-test-slave-4tdj"
server_key: "6cd4336d685e1c529e84210c5e128bb0"
server_key_iv: "1906c4e46e4090ad34a59a181b3c668b"
server_key_version: "encryptionkey@0"
I20260806 06:30:14.891966 14473 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785997581049705-21554-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785997581049705-21554-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785997581049705-21554-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 06:30:14.913326 14473 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 06:30:14.915293 14473 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 06:30:14.915431 14473 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 06:30:14.915679 14473 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 06:30:14.915977 14473 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 06:30:14.916010 14473 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 06:30:14.916041 14473 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 06:30:14.916061 14473 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 06:30:14.921545 14473 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.129:37013
I20260806 06:30:14.921602 14684 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.129:37013 every 8 connection(s)
I20260806 06:30:14.921900 14473 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785997581049705-21554-0/minicluster-data/ts-0/data/info.pb
Aug 06 06:30:14 dist-test-slave-4tdj krb5kdc[14309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997814, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.129@KRBTEST.COM for kudu/127.21.12.190@KRBTEST.COM
I20260806 06:30:14.929211 21554 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHhLhT/build/release/bin/kudu as pid 14473
I20260806 06:30:14.929304 21554 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785997581049705-21554-0/minicluster-data/ts-0/wal/instance
I20260806 06:30:14.929517 21554 external_mini_cluster.cc:1428] Setting key 46fe194742743678b4ae0b267438a19a
I20260806 06:30:14.930708 14440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:14.922630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.129:44045 (local address 127.21.12.190:39233)
0806 06:30:14.922706 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0806 06:30:14.922708 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:30:14.922853 (+ 145us) server_negotiation.cc:407] Connection header received
0806 06:30:14.923363 (+ 510us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:30:14.923364 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:30:14.923378 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:30:14.923417 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:30:14.924334 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:14.924841 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:14.925759 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:14.925944 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:14.925990 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:30:14.927983 (+ 1993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:30:14.927985 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:30:14.927985 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 06:30:14.927991 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:30:14.929087 (+ 1096us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:30:14.929486 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:30:14.929488 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:30:14.929489 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:30:14.929552 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:30:14.929931 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:30:14.929932 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:30:14.929932 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:30:14.930086 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:30:14.930155 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0806 06:30:14.930526 (+ 371us) server_negotiation.cc:299] Negotiation successful
0806 06:30:14.930616 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260806 06:30:14.930840 14691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:14.922594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:39233 (local address 127.21.12.129:44045)
0806 06:30:14.922734 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0806 06:30:14.922771 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0806 06:30:14.923273 (+ 502us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:30:14.923485 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:30:14.923489 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:30:14.924078 (+ 589us) client_negotiation.cc:189] Negotiated authn=SASL
0806 06:30:14.924269 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:30:14.924273 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:30:14.924906 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:30:14.924907 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:30:14.925697 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:30:14.925699 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:30:14.925999 (+ 300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:30:14.926000 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:30:14.926092 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:30:14.926609 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 06:30:14.926618 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0806 06:30:14.927899 (+ 1281us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 06:30:14.929159 (+ 1260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:30:14.929165 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:30:14.929167 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:30:14.929422 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:30:14.929661 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:30:14.929663 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:30:14.929663 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:30:14.929824 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:30:14.930158 (+ 334us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 06:30:14.930166 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 06:30:14.930335 (+ 169us) client_negotiation.cc:769] Sending connection context
0806 06:30:14.930459 (+ 124us) client_negotiation.cc:240] Negotiation successful
0806 06:30:14.930625 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":74,"threads_started":1}
I20260806 06:30:14.932422 14685 heartbeater.cc:347] Connected to a master server at 127.21.12.190:39233
I20260806 06:30:14.932518 14685 heartbeater.cc:464] Registering TS with master...
I20260806 06:30:14.932699 14685 heartbeater.cc:511] Master 127.21.12.190:39233 requested a full tablet report, sending...
I20260806 06:30:14.933197 14375 ts_manager.cc:194] Registered new tserver with Master: b267d2a459e34fefb8d9654d7b63ff45 (127.21.12.129:37013)
I20260806 06:30:14.933779 14375 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.12.129@KRBTEST.COM'} at 127.21.12.129:44045
WARNING: no policy specified for kudu/127.21.12.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.12.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.12.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.130.keytab.
Entry for principal kudu/127.21.12.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.130.keytab.
I20260806 06:30:14.953087 21554 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHhLhT/build/release/bin/kudu
/tmp/dist-test-taskFHhLhT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785997581049705-21554-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785997581049705-21554-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785997581049705-21554-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785997581049705-21554-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.12.130:0
--local_ip_for_outbound_sockets=127.21.12.130
--webserver_interface=127.21.12.130
--webserver_port=0
--tserver_master_addrs=127.21.12.190:39233
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.130.keytab
--principal=kudu/127.21.12.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.148:36879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785997581049705-21554-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785997581049705-21554-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785997581049705-21554-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 06:30:15.034005 14694 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 06:30:15.034197 14694 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 06:30:15.034219 14694 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 06:30:15.035324 14694 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 06:30:15.035363 14694 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 06:30:15.035388 14694 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.12.130
I20260806 06:30:15.036749 14694 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.148:36879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785997581049705-21554-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785997581049705-21554-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.130.keytab
--principal=kudu/127.21.12.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.12.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785997581049705-21554-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.12.130
--webserver_port=0
--tserver_master_addrs=127.21.12.190:39233
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14694
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.12.130
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785997581049705-21554-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 05:11:02 UTC on ccycloud.quasar-hvzbbz.root.comops.site
build id 4569617
I20260806 06:30:15.036939 14694 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 06:30:15.037156 14694 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 06:30:15.041893 14704 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:30:15.041888 14706 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:30:15.041966 14702 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:30:15.042062 14694 server_base.cc:1034] running on GCE node
I20260806 06:30:15.042351 14694 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 06:30:15.042601 14694 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 06:30:15.043741 14694 hybrid_clock.cc:648] HybridClock initialized: now 1785997815043721 us; error 40 us; skew 500 ppm
Aug 06 06:30:15 dist-test-slave-4tdj krb5kdc[14309](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997815, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 06:30:15.046136 14694 init.cc:377] Logged in from keytab as kudu/127.21.12.130@KRBTEST.COM (short username kudu)
I20260806 06:30:15.046607 14694 webserver.cc:460] Webserver started at http://127.21.12.130:33755/ using document root <none> and password file <none>
I20260806 06:30:15.046831 14694 fs_manager.cc:359] Metadata directory not provided
I20260806 06:30:15.046880 14694 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 06:30:15.047000 14694 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 06:30:15.047964 14694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785997581049705-21554-0/minicluster-data/ts-1/data/instance:
uuid: "7d347dcbd0444c33a925274992622695"
format_stamp: "Formatted at 2026-08-06 06:30:15 on dist-test-slave-4tdj"
server_key: "444bd868443ec20276164fc286eaac56"
server_key_iv: "887458ee2755c51b7ba5a3cb862984ad"
server_key_version: "encryptionkey@0"
I20260806 06:30:15.048285 14694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785997581049705-21554-0/minicluster-data/ts-1/wal/instance:
uuid: "7d347dcbd0444c33a925274992622695"
format_stamp: "Formatted at 2026-08-06 06:30:15 on dist-test-slave-4tdj"
server_key: "444bd868443ec20276164fc286eaac56"
server_key_iv: "887458ee2755c51b7ba5a3cb862984ad"
server_key_version: "encryptionkey@0"
I20260806 06:30:15.049485 14694 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260806 06:30:15.050190 14719 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 06:30:15.050388 14694 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260806 06:30:15.050463 14694 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785997581049705-21554-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785997581049705-21554-0/minicluster-data/ts-1/wal
uuid: "7d347dcbd0444c33a925274992622695"
format_stamp: "Formatted at 2026-08-06 06:30:15 on dist-test-slave-4tdj"
server_key: "444bd868443ec20276164fc286eaac56"
server_key_iv: "887458ee2755c51b7ba5a3cb862984ad"
server_key_version: "encryptionkey@0"
I20260806 06:30:15.050513 14694 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785997581049705-21554-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785997581049705-21554-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785997581049705-21554-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 06:30:15.075125 14694 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 06:30:15.076913 14694 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 06:30:15.077029 14694 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 06:30:15.077226 14694 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 06:30:15.077533 14694 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 06:30:15.077567 14694 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 06:30:15.077597 14694 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 06:30:15.077621 14694 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 06:30:15.083221 14694 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.130:40141
I20260806 06:30:15.083264 14920 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.130:40141 every 8 connection(s)
I20260806 06:30:15.083560 14694 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785997581049705-21554-0/minicluster-data/ts-1/data/info.pb
I20260806 06:30:15.087785 21554 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHhLhT/build/release/bin/kudu as pid 14694
I20260806 06:30:15.087919 21554 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785997581049705-21554-0/minicluster-data/ts-1/wal/instance
I20260806 06:30:15.088111 21554 external_mini_cluster.cc:1428] Setting key 6e61f2426e14e8285c3c65e8acc0867c
Aug 06 06:30:15 dist-test-slave-4tdj krb5kdc[14309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997815, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.130@KRBTEST.COM for kudu/127.21.12.190@KRBTEST.COM
I20260806 06:30:15.092739 14440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:15.084257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.130:44315 (local address 127.21.12.190:39233)
0806 06:30:15.084315 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0806 06:30:15.084317 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:30:15.084495 (+ 178us) server_negotiation.cc:407] Connection header received
0806 06:30:15.084998 (+ 503us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:30:15.085000 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:30:15.085012 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:30:15.085061 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:30:15.086006 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:15.086559 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:15.087508 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:15.087756 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:15.087823 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:30:15.090132 (+ 2309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:30:15.090134 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:30:15.090134 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 06:30:15.090141 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:30:15.091456 (+ 1315us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:30:15.091910 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:30:15.091910 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:30:15.091911 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:30:15.091958 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:30:15.092194 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:30:15.092195 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:30:15.092195 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:30:15.092287 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:30:15.092330 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0806 06:30:15.092608 (+ 278us) server_negotiation.cc:299] Negotiation successful
0806 06:30:15.092675 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260806 06:30:15.092975 14926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:15.084226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:39233 (local address 127.21.12.130:44315)
0806 06:30:15.084384 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0806 06:30:15.084420 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0806 06:30:15.084909 (+ 489us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:30:15.085141 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:30:15.085144 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:30:15.085729 (+ 585us) client_negotiation.cc:189] Negotiated authn=SASL
0806 06:30:15.085916 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:30:15.085921 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:30:15.086642 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:30:15.086643 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:30:15.087426 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:30:15.087428 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:30:15.087863 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:30:15.087864 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:30:15.088018 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:30:15.088444 (+ 426us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 06:30:15.088452 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0806 06:30:15.090031 (+ 1579us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 06:30:15.091534 (+ 1503us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:30:15.091541 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:30:15.091543 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:30:15.091841 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:30:15.092010 (+ 169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:30:15.092011 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:30:15.092011 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:30:15.092139 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:30:15.092335 (+ 196us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 06:30:15.092340 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 06:30:15.092491 (+ 151us) client_negotiation.cc:769] Sending connection context
0806 06:30:15.092607 (+ 116us) client_negotiation.cc:240] Negotiation successful
0806 06:30:15.092765 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":81,"threads_started":1}
I20260806 06:30:15.093482 14921 heartbeater.cc:347] Connected to a master server at 127.21.12.190:39233
I20260806 06:30:15.093580 14921 heartbeater.cc:464] Registering TS with master...
I20260806 06:30:15.093762 14921 heartbeater.cc:511] Master 127.21.12.190:39233 requested a full tablet report, sending...
I20260806 06:30:15.094146 14375 ts_manager.cc:194] Registered new tserver with Master: 7d347dcbd0444c33a925274992622695 (127.21.12.130:40141)
I20260806 06:30:15.094597 14375 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.12.130@KRBTEST.COM'} at 127.21.12.130:44315
WARNING: no policy specified for kudu/127.21.12.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.12.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.12.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.131.keytab.
Entry for principal kudu/127.21.12.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.131.keytab.
I20260806 06:30:15.112780 21554 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHhLhT/build/release/bin/kudu
/tmp/dist-test-taskFHhLhT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785997581049705-21554-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785997581049705-21554-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785997581049705-21554-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785997581049705-21554-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.12.131:0
--local_ip_for_outbound_sockets=127.21.12.131
--webserver_interface=127.21.12.131
--webserver_port=0
--tserver_master_addrs=127.21.12.190:39233
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.131.keytab
--principal=kudu/127.21.12.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.148:36879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785997581049705-21554-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785997581049705-21554-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785997581049705-21554-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 06:30:15.197011 14935 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 06:30:15.197202 14935 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 06:30:15.197224 14935 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 06:30:15.198329 14935 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 06:30:15.198360 14935 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 06:30:15.198385 14935 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.12.131
I20260806 06:30:15.199800 14935 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.148:36879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785997581049705-21554-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785997581049705-21554-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.131.keytab
--principal=kudu/127.21.12.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.12.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785997581049705-21554-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.12.131
--webserver_port=0
--tserver_master_addrs=127.21.12.190:39233
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14935
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.12.131
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785997581049705-21554-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 05:11:02 UTC on ccycloud.quasar-hvzbbz.root.comops.site
build id 4569617
I20260806 06:30:15.199980 14935 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 06:30:15.200155 14935 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 06:30:15.204834 14942 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:30:15.204906 14943 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:30:15.204838 14945 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:30:15.205188 14935 server_base.cc:1034] running on GCE node
I20260806 06:30:15.205366 14935 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 06:30:15.205550 14935 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 06:30:15.206681 14935 hybrid_clock.cc:648] HybridClock initialized: now 1785997815206660 us; error 34 us; skew 500 ppm
Aug 06 06:30:15 dist-test-slave-4tdj krb5kdc[14309](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997815, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 06:30:15.209046 14935 init.cc:377] Logged in from keytab as kudu/127.21.12.131@KRBTEST.COM (short username kudu)
I20260806 06:30:15.209460 14935 webserver.cc:460] Webserver started at http://127.21.12.131:45833/ using document root <none> and password file <none>
I20260806 06:30:15.209653 14935 fs_manager.cc:359] Metadata directory not provided
I20260806 06:30:15.209699 14935 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 06:30:15.209820 14935 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 06:30:15.210868 14935 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785997581049705-21554-0/minicluster-data/ts-2/data/instance:
uuid: "911a0150a106481dbe2ce87e3c95f546"
format_stamp: "Formatted at 2026-08-06 06:30:15 on dist-test-slave-4tdj"
server_key: "08b3391c5c8c4c80126fce6bd5989856"
server_key_iv: "7fb721284f8269534f19cd242f197604"
server_key_version: "encryptionkey@0"
I20260806 06:30:15.211200 14935 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785997581049705-21554-0/minicluster-data/ts-2/wal/instance:
uuid: "911a0150a106481dbe2ce87e3c95f546"
format_stamp: "Formatted at 2026-08-06 06:30:15 on dist-test-slave-4tdj"
server_key: "08b3391c5c8c4c80126fce6bd5989856"
server_key_iv: "7fb721284f8269534f19cd242f197604"
server_key_version: "encryptionkey@0"
I20260806 06:30:15.212383 14935 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260806 06:30:15.213030 14957 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 06:30:15.213184 14935 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260806 06:30:15.213267 14935 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785997581049705-21554-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785997581049705-21554-0/minicluster-data/ts-2/wal
uuid: "911a0150a106481dbe2ce87e3c95f546"
format_stamp: "Formatted at 2026-08-06 06:30:15 on dist-test-slave-4tdj"
server_key: "08b3391c5c8c4c80126fce6bd5989856"
server_key_iv: "7fb721284f8269534f19cd242f197604"
server_key_version: "encryptionkey@0"
I20260806 06:30:15.213330 14935 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785997581049705-21554-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785997581049705-21554-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785997581049705-21554-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 06:30:15.238809 14935 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 06:30:15.240918 14935 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 06:30:15.241058 14935 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 06:30:15.241289 14935 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 06:30:15.241590 14935 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 06:30:15.241626 14935 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 06:30:15.241647 14935 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 06:30:15.241660 14935 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 06:30:15.247367 14935 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.131:42901
I20260806 06:30:15.247442 15130 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.131:42901 every 8 connection(s)
I20260806 06:30:15.247709 14935 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785997581049705-21554-0/minicluster-data/ts-2/data/info.pb
Aug 06 06:30:15 dist-test-slave-4tdj krb5kdc[14309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997815, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.131@KRBTEST.COM for kudu/127.21.12.190@KRBTEST.COM
I20260806 06:30:15.256279 14440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:15.248351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.131:49439 (local address 127.21.12.190:39233)
0806 06:30:15.248403 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0806 06:30:15.248405 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:30:15.248593 (+ 188us) server_negotiation.cc:407] Connection header received
0806 06:30:15.249167 (+ 574us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:30:15.249168 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:30:15.249212 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:30:15.249270 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:30:15.250234 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:15.250892 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:15.251838 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:15.252018 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:15.252065 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:30:15.253974 (+ 1909us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:30:15.253976 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:30:15.253977 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 06:30:15.253986 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:30:15.255248 (+ 1262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:30:15.255577 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:30:15.255578 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:30:15.255578 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:30:15.255623 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:30:15.255809 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:30:15.255809 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:30:15.255809 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:30:15.255900 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:30:15.255944 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0806 06:30:15.256143 (+ 199us) server_negotiation.cc:299] Negotiation successful
0806 06:30:15.256214 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260806 06:30:15.256403 15134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:15.248325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:39233 (local address 127.21.12.131:49439)
0806 06:30:15.248481 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0806 06:30:15.248517 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0806 06:30:15.249051 (+ 534us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:30:15.249340 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:30:15.249345 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:30:15.249955 (+ 610us) client_negotiation.cc:189] Negotiated authn=SASL
0806 06:30:15.250156 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:30:15.250161 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:30:15.250972 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:30:15.250973 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:30:15.251775 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:30:15.251777 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:30:15.252073 (+ 296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:30:15.252074 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:30:15.252174 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:30:15.252575 (+ 401us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 06:30:15.252583 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0806 06:30:15.253870 (+ 1287us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 06:30:15.255319 (+ 1449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:30:15.255324 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:30:15.255327 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:30:15.255525 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:30:15.255666 (+ 141us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:30:15.255666 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:30:15.255667 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:30:15.255762 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:30:15.255939 (+ 177us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 06:30:15.255942 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 06:30:15.256043 (+ 101us) client_negotiation.cc:769] Sending connection context
0806 06:30:15.256124 (+ 81us) client_negotiation.cc:240] Negotiation successful
0806 06:30:15.256234 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":76,"threads_started":1}
I20260806 06:30:15.256870 15131 heartbeater.cc:347] Connected to a master server at 127.21.12.190:39233
I20260806 06:30:15.256958 15131 heartbeater.cc:464] Registering TS with master...
I20260806 06:30:15.257131 15131 heartbeater.cc:511] Master 127.21.12.190:39233 requested a full tablet report, sending...
I20260806 06:30:15.257488 14375 ts_manager.cc:194] Registered new tserver with Master: 911a0150a106481dbe2ce87e3c95f546 (127.21.12.131:42901)
I20260806 06:30:15.257691 21554 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHhLhT/build/release/bin/kudu as pid 14935
I20260806 06:30:15.257802 21554 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785997581049705-21554-0/minicluster-data/ts-2/wal/instance
I20260806 06:30:15.257992 14375 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.12.131@KRBTEST.COM'} at 127.21.12.131:49439
I20260806 06:30:15.258011 21554 external_mini_cluster.cc:1428] Setting key 2299133676a666aa3845e441ffb2b27c
I20260806 06:30:15.258837 21554 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 06:30:15.265246 14440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:15.260120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49810 (local address 127.21.12.190:39233)
0806 06:30:15.260185 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0806 06:30:15.260187 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:30:15.260276 (+ 89us) server_negotiation.cc:407] Connection header received
0806 06:30:15.260336 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:30:15.260337 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:30:15.260346 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:30:15.260390 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:30:15.260927 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:15.261377 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:15.262074 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:15.262259 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:15.262331 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:30:15.263292 (+ 961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:30:15.263293 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:30:15.263293 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 06:30:15.263301 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:30:15.264238 (+ 937us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:30:15.264526 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:30:15.264527 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:30:15.264528 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:30:15.264578 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:30:15.264745 (+ 167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:30:15.264746 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:30:15.264747 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:30:15.264879 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:30:15.264965 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0806 06:30:15.265066 (+ 101us) server_negotiation.cc:299] Negotiation successful
0806 06:30:15.265140 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260806 06:30:15.266556 14375 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49810:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 06:30:15.268100 14375 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 06:30:15.274323 15154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:15.270419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.130:40141 (local address 127.0.0.1:45306)
0806 06:30:15.270548 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0806 06:30:15.270557 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0806 06:30:15.270599 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:30:15.271150 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:30:15.271151 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:30:15.271153 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 06:30:15.271195 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:30:15.271197 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:30:15.272315 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:30:15.272316 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:30:15.273073 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:30:15.273076 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:30:15.273990 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:30:15.273991 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:30:15.274068 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:30:15.274080 (+ 12us) client_negotiation.cc:769] Sending connection context
0806 06:30:15.274170 (+ 90us) client_negotiation.cc:240] Negotiation successful
0806 06:30:15.274224 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":73,"threads_started":1}
I20260806 06:30:15.274329 15155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:15.270626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45306 (local address 127.21.12.130:40141)
0806 06:30:15.270793 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0806 06:30:15.270797 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 06:30:15.270808 (+ 11us) server_negotiation.cc:407] Connection header received
0806 06:30:15.270847 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:30:15.270849 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:30:15.270865 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:30:15.270954 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 06:30:15.271277 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:15.272234 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:15.273169 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:15.273879 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:15.273958 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:30:15.274013 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0806 06:30:15.274180 (+ 167us) server_negotiation.cc:299] Negotiation successful
0806 06:30:15.274223 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":83,"threads_started":1}
I20260806 06:30:15.274650 15152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:15.270324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41080 (local address 127.21.12.131:42901)
0806 06:30:15.270625 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0806 06:30:15.270630 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 06:30:15.270639 (+ 9us) server_negotiation.cc:407] Connection header received
0806 06:30:15.270671 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:30:15.270673 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:30:15.270687 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:30:15.270754 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 06:30:15.270897 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:15.271920 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:15.273416 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:15.274089 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:15.274155 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:30:15.274207 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0806 06:30:15.274526 (+ 319us) server_negotiation.cc:299] Negotiation successful
0806 06:30:15.274570 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261,"thread_start_us":68,"threads_started":1}
I20260806 06:30:15.274444 15149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:15.270223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.131:42901 (local address 127.0.0.1:41080)
0806 06:30:15.270495 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0806 06:30:15.270505 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0806 06:30:15.270557 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:30:15.270773 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:30:15.270776 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:30:15.270778 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 06:30:15.270837 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:30:15.270838 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:30:15.272014 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:30:15.272015 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:30:15.273340 (+ 1325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:30:15.273341 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:30:15.274185 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:30:15.274186 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:30:15.274244 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:30:15.274248 (+ 4us) client_negotiation.cc:769] Sending connection context
0806 06:30:15.274335 (+ 87us) client_negotiation.cc:240] Negotiation successful
0806 06:30:15.274376 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"mutex_wait_us":51,"thread_start_us":97,"threads_started":1}
I20260806 06:30:15.275170 15151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:15.270207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.129:37013 (local address 127.0.0.1:35780)
0806 06:30:15.270389 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0806 06:30:15.270428 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0806 06:30:15.270489 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:30:15.270972 (+ 483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:30:15.270973 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:30:15.270975 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 06:30:15.271015 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:30:15.271016 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:30:15.271960 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:30:15.271961 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:30:15.273324 (+ 1363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:30:15.273328 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:30:15.274920 (+ 1592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:30:15.274922 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:30:15.274987 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:30:15.274992 (+ 5us) client_negotiation.cc:769] Sending connection context
0806 06:30:15.275056 (+ 64us) client_negotiation.cc:240] Negotiation successful
0806 06:30:15.275100 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"mutex_wait_us":88,"thread_start_us":55,"threads_started":1}
I20260806 06:30:15.275422 15153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:15.270334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35780 (local address 127.21.12.129:37013)
0806 06:30:15.270837 (+ 503us) server_negotiation.cc:206] Beginning negotiation
0806 06:30:15.270841 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 06:30:15.270849 (+ 8us) server_negotiation.cc:407] Connection header received
0806 06:30:15.270884 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:30:15.270887 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:30:15.270901 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:30:15.270970 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 06:30:15.271071 (+ 101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:15.271862 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:15.274245 (+ 2383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:15.274847 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:15.274918 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:30:15.274977 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0806 06:30:15.275294 (+ 317us) server_negotiation.cc:299] Negotiation successful
0806 06:30:15.275343 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":462,"thread_start_us":73,"threads_started":1}
I20260806 06:30:15.275416 14805 tablet_service.cc:1467] Processing CreateTablet for tablet 1344bd342d224882a2b3babafbb1e995 (DEFAULT_TABLE table=test-table [id=57ecafca3b534c42aeb6bdf1a442c29f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 06:30:15.275571 15026 tablet_service.cc:1467] Processing CreateTablet for tablet 1344bd342d224882a2b3babafbb1e995 (DEFAULT_TABLE table=test-table [id=57ecafca3b534c42aeb6bdf1a442c29f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 06:30:15.275781 14805 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1344bd342d224882a2b3babafbb1e995. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 06:30:15.275898 15026 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1344bd342d224882a2b3babafbb1e995. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 06:30:15.278229 15158 tablet_bootstrap.cc:492] T 1344bd342d224882a2b3babafbb1e995 P 7d347dcbd0444c33a925274992622695: Bootstrap starting.
I20260806 06:30:15.278563 15157 tablet_bootstrap.cc:492] T 1344bd342d224882a2b3babafbb1e995 P 911a0150a106481dbe2ce87e3c95f546: Bootstrap starting.
I20260806 06:30:15.278873 14582 tablet_service.cc:1467] Processing CreateTablet for tablet 1344bd342d224882a2b3babafbb1e995 (DEFAULT_TABLE table=test-table [id=57ecafca3b534c42aeb6bdf1a442c29f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 06:30:15.279124 15157 tablet_bootstrap.cc:654] T 1344bd342d224882a2b3babafbb1e995 P 911a0150a106481dbe2ce87e3c95f546: Neither blocks nor log segments found. Creating new log.
I20260806 06:30:15.279150 15158 tablet_bootstrap.cc:654] T 1344bd342d224882a2b3babafbb1e995 P 7d347dcbd0444c33a925274992622695: Neither blocks nor log segments found. Creating new log.
I20260806 06:30:15.279241 14582 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1344bd342d224882a2b3babafbb1e995. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 06:30:15.279587 15157 log.cc:824] T 1344bd342d224882a2b3babafbb1e995 P 911a0150a106481dbe2ce87e3c95f546: Log is configured to *not* fsync() on all Append() calls
I20260806 06:30:15.279760 15158 log.cc:824] T 1344bd342d224882a2b3babafbb1e995 P 7d347dcbd0444c33a925274992622695: Log is configured to *not* fsync() on all Append() calls
I20260806 06:30:15.280264 15157 tablet_bootstrap.cc:492] T 1344bd342d224882a2b3babafbb1e995 P 911a0150a106481dbe2ce87e3c95f546: No bootstrap required, opened a new log
I20260806 06:30:15.280329 15157 ts_tablet_manager.cc:1397] T 1344bd342d224882a2b3babafbb1e995 P 911a0150a106481dbe2ce87e3c95f546: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260806 06:30:15.280545 15158 tablet_bootstrap.cc:492] T 1344bd342d224882a2b3babafbb1e995 P 7d347dcbd0444c33a925274992622695: No bootstrap required, opened a new log
I20260806 06:30:15.280628 15158 ts_tablet_manager.cc:1397] T 1344bd342d224882a2b3babafbb1e995 P 7d347dcbd0444c33a925274992622695: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260806 06:30:15.281754 15157 raft_consensus.cc:348] T 1344bd342d224882a2b3babafbb1e995 P 911a0150a106481dbe2ce87e3c95f546 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b267d2a459e34fefb8d9654d7b63ff45" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 37013 } } peers { permanent_uuid: "7d347dcbd0444c33a925274992622695" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 40141 } } peers { permanent_uuid: "911a0150a106481dbe2ce87e3c95f546" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 42901 } }
I20260806 06:30:15.281934 15157 raft_consensus.cc:374] T 1344bd342d224882a2b3babafbb1e995 P 911a0150a106481dbe2ce87e3c95f546 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 06:30:15.281996 15157 raft_consensus.cc:729] T 1344bd342d224882a2b3babafbb1e995 P 911a0150a106481dbe2ce87e3c95f546 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 911a0150a106481dbe2ce87e3c95f546, State: Initialized, Role: FOLLOWER
I20260806 06:30:15.282115 15157 consensus_queue.cc:260] T 1344bd342d224882a2b3babafbb1e995 P 911a0150a106481dbe2ce87e3c95f546 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b267d2a459e34fefb8d9654d7b63ff45" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 37013 } } peers { permanent_uuid: "7d347dcbd0444c33a925274992622695" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 40141 } } peers { permanent_uuid: "911a0150a106481dbe2ce87e3c95f546" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 42901 } }
I20260806 06:30:15.282416 15131 heartbeater.cc:503] Master 127.21.12.190:39233 was elected leader, sending a full tablet report...
I20260806 06:30:15.282401 15157 ts_tablet_manager.cc:1428] T 1344bd342d224882a2b3babafbb1e995 P 911a0150a106481dbe2ce87e3c95f546: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260806 06:30:15.282693 15164 tablet_bootstrap.cc:492] T 1344bd342d224882a2b3babafbb1e995 P b267d2a459e34fefb8d9654d7b63ff45: Bootstrap starting.
I20260806 06:30:15.282788 15158 raft_consensus.cc:348] T 1344bd342d224882a2b3babafbb1e995 P 7d347dcbd0444c33a925274992622695 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b267d2a459e34fefb8d9654d7b63ff45" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 37013 } } peers { permanent_uuid: "7d347dcbd0444c33a925274992622695" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 40141 } } peers { permanent_uuid: "911a0150a106481dbe2ce87e3c95f546" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 42901 } }
I20260806 06:30:15.282991 15158 raft_consensus.cc:374] T 1344bd342d224882a2b3babafbb1e995 P 7d347dcbd0444c33a925274992622695 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 06:30:15.283085 15158 raft_consensus.cc:729] T 1344bd342d224882a2b3babafbb1e995 P 7d347dcbd0444c33a925274992622695 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d347dcbd0444c33a925274992622695, State: Initialized, Role: FOLLOWER
I20260806 06:30:15.283215 15158 consensus_queue.cc:260] T 1344bd342d224882a2b3babafbb1e995 P 7d347dcbd0444c33a925274992622695 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b267d2a459e34fefb8d9654d7b63ff45" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 37013 } } peers { permanent_uuid: "7d347dcbd0444c33a925274992622695" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 40141 } } peers { permanent_uuid: "911a0150a106481dbe2ce87e3c95f546" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 42901 } }
I20260806 06:30:15.283404 15164 tablet_bootstrap.cc:654] T 1344bd342d224882a2b3babafbb1e995 P b267d2a459e34fefb8d9654d7b63ff45: Neither blocks nor log segments found. Creating new log.
I20260806 06:30:15.283569 14921 heartbeater.cc:503] Master 127.21.12.190:39233 was elected leader, sending a full tablet report...
I20260806 06:30:15.283571 15158 ts_tablet_manager.cc:1428] T 1344bd342d224882a2b3babafbb1e995 P 7d347dcbd0444c33a925274992622695: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260806 06:30:15.284022 15164 log.cc:824] T 1344bd342d224882a2b3babafbb1e995 P b267d2a459e34fefb8d9654d7b63ff45: Log is configured to *not* fsync() on all Append() calls
I20260806 06:30:15.284811 15164 tablet_bootstrap.cc:492] T 1344bd342d224882a2b3babafbb1e995 P b267d2a459e34fefb8d9654d7b63ff45: No bootstrap required, opened a new log
I20260806 06:30:15.284873 15164 ts_tablet_manager.cc:1397] T 1344bd342d224882a2b3babafbb1e995 P b267d2a459e34fefb8d9654d7b63ff45: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260806 06:30:15.286381 15164 raft_consensus.cc:348] T 1344bd342d224882a2b3babafbb1e995 P b267d2a459e34fefb8d9654d7b63ff45 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b267d2a459e34fefb8d9654d7b63ff45" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 37013 } } peers { permanent_uuid: "7d347dcbd0444c33a925274992622695" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 40141 } } peers { permanent_uuid: "911a0150a106481dbe2ce87e3c95f546" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 42901 } }
I20260806 06:30:15.286527 15164 raft_consensus.cc:374] T 1344bd342d224882a2b3babafbb1e995 P b267d2a459e34fefb8d9654d7b63ff45 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 06:30:15.286551 15164 raft_consensus.cc:729] T 1344bd342d224882a2b3babafbb1e995 P b267d2a459e34fefb8d9654d7b63ff45 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b267d2a459e34fefb8d9654d7b63ff45, State: Initialized, Role: FOLLOWER
I20260806 06:30:15.286638 15164 consensus_queue.cc:260] T 1344bd342d224882a2b3babafbb1e995 P b267d2a459e34fefb8d9654d7b63ff45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b267d2a459e34fefb8d9654d7b63ff45" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 37013 } } peers { permanent_uuid: "7d347dcbd0444c33a925274992622695" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 40141 } } peers { permanent_uuid: "911a0150a106481dbe2ce87e3c95f546" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 42901 } }
I20260806 06:30:15.286864 15164 ts_tablet_manager.cc:1428] T 1344bd342d224882a2b3babafbb1e995 P b267d2a459e34fefb8d9654d7b63ff45: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260806 06:30:15.286896 14685 heartbeater.cc:503] Master 127.21.12.190:39233 was elected leader, sending a full tablet report...
W20260806 06:30:15.334100 14924 tablet.cc:2369] T 1344bd342d224882a2b3babafbb1e995 P 7d347dcbd0444c33a925274992622695: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 06:30:15.422632 14688 tablet.cc:2369] T 1344bd342d224882a2b3babafbb1e995 P b267d2a459e34fefb8d9654d7b63ff45: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 06:30:15.457221 15166 raft_consensus.cc:482] T 1344bd342d224882a2b3babafbb1e995 P 7d347dcbd0444c33a925274992622695 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 06:30:15.457363 15166 raft_consensus.cc:504] T 1344bd342d224882a2b3babafbb1e995 P 7d347dcbd0444c33a925274992622695 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b267d2a459e34fefb8d9654d7b63ff45" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 37013 } } peers { permanent_uuid: "7d347dcbd0444c33a925274992622695" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 40141 } } peers { permanent_uuid: "911a0150a106481dbe2ce87e3c95f546" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 42901 } }
I20260806 06:30:15.457736 15166 leader_election.cc:288] T 1344bd342d224882a2b3babafbb1e995 P 7d347dcbd0444c33a925274992622695 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b267d2a459e34fefb8d9654d7b63ff45 (127.21.12.129:37013), 911a0150a106481dbe2ce87e3c95f546 (127.21.12.131:42901)
I20260806 06:30:15.460637 15153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:15.458002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.130:44939 (local address 127.21.12.129:37013)
0806 06:30:15.458046 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0806 06:30:15.458048 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:30:15.458057 (+ 9us) server_negotiation.cc:407] Connection header received
0806 06:30:15.458086 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:30:15.458087 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:30:15.458095 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:30:15.458162 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 06:30:15.458359 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:15.458881 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:15.459900 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:15.460317 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:15.460371 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:30:15.460378 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0806 06:30:15.460534 (+ 156us) server_negotiation.cc:299] Negotiation successful
0806 06:30:15.460571 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260806 06:30:15.460637 14926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:15.457892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.129:37013 (local address 127.21.12.130:44939)
0806 06:30:15.457935 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0806 06:30:15.457946 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0806 06:30:15.457986 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:30:15.458183 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:30:15.458184 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:30:15.458186 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 06:30:15.458256 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:30:15.458258 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:30:15.458948 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:30:15.458949 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:30:15.459819 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:30:15.459822 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:30:15.460390 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:30:15.460391 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:30:15.460470 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:30:15.460473 (+ 3us) client_negotiation.cc:769] Sending connection context
0806 06:30:15.460516 (+ 43us) client_negotiation.cc:240] Negotiation successful
0806 06:30:15.460550 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21}
I20260806 06:30:15.460668 15152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:15.457979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.130:41127 (local address 127.21.12.131:42901)
0806 06:30:15.458033 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0806 06:30:15.458035 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:30:15.458261 (+ 226us) server_negotiation.cc:407] Connection header received
0806 06:30:15.458315 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:30:15.458316 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:30:15.458327 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:30:15.458378 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 06:30:15.458521 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:15.459021 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:15.459935 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:15.460369 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:15.460441 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:30:15.460450 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0806 06:30:15.460575 (+ 125us) server_negotiation.cc:299] Negotiation successful
0806 06:30:15.460612 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260806 06:30:15.460721 15169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:15.457894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.131:42901 (local address 127.21.12.130:41127)
0806 06:30:15.458196 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0806 06:30:15.458205 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0806 06:30:15.458255 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:30:15.458392 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:30:15.458392 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:30:15.458394 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 06:30:15.458460 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:30:15.458462 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:30:15.459100 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:30:15.459101 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:30:15.459852 (+ 751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:30:15.459855 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:30:15.460448 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:30:15.460449 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:30:15.460502 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:30:15.460507 (+ 5us) client_negotiation.cc:769] Sending connection context
0806 06:30:15.460559 (+ 52us) client_negotiation.cc:240] Negotiation successful
0806 06:30:15.460601 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"mutex_wait_us":140,"thread_start_us":99,"threads_started":1}
I20260806 06:30:15.460976 14613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1344bd342d224882a2b3babafbb1e995" candidate_uuid: "7d347dcbd0444c33a925274992622695" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b267d2a459e34fefb8d9654d7b63ff45" is_pre_election: true
I20260806 06:30:15.461134 14613 raft_consensus.cc:2454] T 1344bd342d224882a2b3babafbb1e995 P b267d2a459e34fefb8d9654d7b63ff45 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d347dcbd0444c33a925274992622695 in term 0.
I20260806 06:30:15.461215 15055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1344bd342d224882a2b3babafbb1e995" candidate_uuid: "7d347dcbd0444c33a925274992622695" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "911a0150a106481dbe2ce87e3c95f546" is_pre_election: true
I20260806 06:30:15.461319 15055 raft_consensus.cc:2454] T 1344bd342d224882a2b3babafbb1e995 P 911a0150a106481dbe2ce87e3c95f546 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d347dcbd0444c33a925274992622695 in term 0.
I20260806 06:30:15.461369 14722 leader_election.cc:302] T 1344bd342d224882a2b3babafbb1e995 P 7d347dcbd0444c33a925274992622695 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7d347dcbd0444c33a925274992622695, b267d2a459e34fefb8d9654d7b63ff45; no voters:
I20260806 06:30:15.461493 15166 raft_consensus.cc:2781] T 1344bd342d224882a2b3babafbb1e995 P 7d347dcbd0444c33a925274992622695 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 06:30:15.461552 15166 raft_consensus.cc:482] T 1344bd342d224882a2b3babafbb1e995 P 7d347dcbd0444c33a925274992622695 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 06:30:15.461575 15166 raft_consensus.cc:3037] T 1344bd342d224882a2b3babafbb1e995 P 7d347dcbd0444c33a925274992622695 [term 0 FOLLOWER]: Advancing to term 1
I20260806 06:30:15.462249 15166 raft_consensus.cc:504] T 1344bd342d224882a2b3babafbb1e995 P 7d347dcbd0444c33a925274992622695 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b267d2a459e34fefb8d9654d7b63ff45" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 37013 } } peers { permanent_uuid: "7d347dcbd0444c33a925274992622695" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 40141 } } peers { permanent_uuid: "911a0150a106481dbe2ce87e3c95f546" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 42901 } }
I20260806 06:30:15.462390 15166 leader_election.cc:288] T 1344bd342d224882a2b3babafbb1e995 P 7d347dcbd0444c33a925274992622695 [CANDIDATE]: Term 1 election: Requested vote from peers b267d2a459e34fefb8d9654d7b63ff45 (127.21.12.129:37013), 911a0150a106481dbe2ce87e3c95f546 (127.21.12.131:42901)
I20260806 06:30:15.462548 14613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1344bd342d224882a2b3babafbb1e995" candidate_uuid: "7d347dcbd0444c33a925274992622695" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b267d2a459e34fefb8d9654d7b63ff45"
I20260806 06:30:15.462574 15055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1344bd342d224882a2b3babafbb1e995" candidate_uuid: "7d347dcbd0444c33a925274992622695" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "911a0150a106481dbe2ce87e3c95f546"
I20260806 06:30:15.462630 14613 raft_consensus.cc:3037] T 1344bd342d224882a2b3babafbb1e995 P b267d2a459e34fefb8d9654d7b63ff45 [term 0 FOLLOWER]: Advancing to term 1
I20260806 06:30:15.462644 15055 raft_consensus.cc:3037] T 1344bd342d224882a2b3babafbb1e995 P 911a0150a106481dbe2ce87e3c95f546 [term 0 FOLLOWER]: Advancing to term 1
I20260806 06:30:15.463289 14613 raft_consensus.cc:2454] T 1344bd342d224882a2b3babafbb1e995 P b267d2a459e34fefb8d9654d7b63ff45 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d347dcbd0444c33a925274992622695 in term 1.
I20260806 06:30:15.463376 15055 raft_consensus.cc:2454] T 1344bd342d224882a2b3babafbb1e995 P 911a0150a106481dbe2ce87e3c95f546 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d347dcbd0444c33a925274992622695 in term 1.
I20260806 06:30:15.463449 14722 leader_election.cc:302] T 1344bd342d224882a2b3babafbb1e995 P 7d347dcbd0444c33a925274992622695 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7d347dcbd0444c33a925274992622695, b267d2a459e34fefb8d9654d7b63ff45; no voters:
I20260806 06:30:15.463542 15166 raft_consensus.cc:2781] T 1344bd342d224882a2b3babafbb1e995 P 7d347dcbd0444c33a925274992622695 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 06:30:15.463701 15166 raft_consensus.cc:686] T 1344bd342d224882a2b3babafbb1e995 P 7d347dcbd0444c33a925274992622695 [term 1 LEADER]: Becoming Leader. State: Replica: 7d347dcbd0444c33a925274992622695, State: Running, Role: LEADER
I20260806 06:30:15.463793 15166 consensus_queue.cc:237] T 1344bd342d224882a2b3babafbb1e995 P 7d347dcbd0444c33a925274992622695 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b267d2a459e34fefb8d9654d7b63ff45" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 37013 } } peers { permanent_uuid: "7d347dcbd0444c33a925274992622695" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 40141 } } peers { permanent_uuid: "911a0150a106481dbe2ce87e3c95f546" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 42901 } }
I20260806 06:30:15.464576 14375 catalog_manager.cc:5495] T 1344bd342d224882a2b3babafbb1e995 P 7d347dcbd0444c33a925274992622695 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7d347dcbd0444c33a925274992622695 (127.21.12.130). New cstate: current_term: 1 leader_uuid: "7d347dcbd0444c33a925274992622695" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b267d2a459e34fefb8d9654d7b63ff45" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 37013 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7d347dcbd0444c33a925274992622695" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 40141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "911a0150a106481dbe2ce87e3c95f546" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 42901 } health_report { overall_health: UNKNOWN } } }
I20260806 06:30:15.497939 15155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:15.495960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45318 (local address 127.21.12.130:40141)
0806 06:30:15.496008 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0806 06:30:15.496010 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:30:15.496019 (+ 9us) server_negotiation.cc:407] Connection header received
0806 06:30:15.496056 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:30:15.496056 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:30:15.496075 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:30:15.496135 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 06:30:15.496240 (+ 105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:15.496695 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:15.497370 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:15.497540 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:15.497591 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:30:15.497683 (+ 92us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 06:30:15.497782 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 06:30:15.497837 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0806 06:30:15.497868 (+ 31us) server_negotiation.cc:299] Negotiation successful
0806 06:30:15.497883 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
W20260806 06:30:15.498222 15132 tablet.cc:2369] T 1344bd342d224882a2b3babafbb1e995 P 911a0150a106481dbe2ce87e3c95f546: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 06:30:15.499810 15055 raft_consensus.cc:1261] T 1344bd342d224882a2b3babafbb1e995 P 911a0150a106481dbe2ce87e3c95f546 [term 1 FOLLOWER]: Refusing update from remote peer 7d347dcbd0444c33a925274992622695: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 06:30:15.499811 14613 raft_consensus.cc:1261] T 1344bd342d224882a2b3babafbb1e995 P b267d2a459e34fefb8d9654d7b63ff45 [term 1 FOLLOWER]: Refusing update from remote peer 7d347dcbd0444c33a925274992622695: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 06:30:15.500145 15166 consensus_queue.cc:1035] T 1344bd342d224882a2b3babafbb1e995 P 7d347dcbd0444c33a925274992622695 [LEADER]: Connected to new peer: Peer: permanent_uuid: "911a0150a106481dbe2ce87e3c95f546" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 42901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 06:30:15.500303 15172 consensus_queue.cc:1035] T 1344bd342d224882a2b3babafbb1e995 P 7d347dcbd0444c33a925274992622695 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b267d2a459e34fefb8d9654d7b63ff45" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 37013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 06:30:15.502480 15179 mvcc.cc:205] Tried to move back new op lower bound from 7315447052284436480 to 7315447052140105728. Current Snapshot: MvccSnapshot[applied={T|T < 7315447052284436480}]
I20260806 06:30:15.502619 15181 mvcc.cc:205] Tried to move back new op lower bound from 7315447052284436480 to 7315447052140105728. Current Snapshot: MvccSnapshot[applied={T|T < 7315447052284436480}]
I20260806 06:30:15.502737 15180 mvcc.cc:205] Tried to move back new op lower bound from 7315447052284436480 to 7315447052140105728. Current Snapshot: MvccSnapshot[applied={T|T < 7315447052284436480}]
I20260806 06:30:15.504688 14374 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49810:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 06:30:15.504805 14374 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49810:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 06:30:15.505606 14374 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6f886f0db3be4ddfb6f3113cab9f2a79: Sending DeleteTablet for 3 replicas of tablet 1344bd342d224882a2b3babafbb1e995
I20260806 06:30:15.505921 15026 tablet_service.cc:1514] Processing DeleteTablet for tablet 1344bd342d224882a2b3babafbb1e995 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 06:30:15 UTC) from {username='kudu', principal='kudu/127.21.12.190@KRBTEST.COM'} at 127.0.0.1:41080
I20260806 06:30:15.505957 14805 tablet_service.cc:1514] Processing DeleteTablet for tablet 1344bd342d224882a2b3babafbb1e995 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 06:30:15 UTC) from {username='kudu', principal='kudu/127.21.12.190@KRBTEST.COM'} at 127.0.0.1:45306
I20260806 06:30:15.505995 14582 tablet_service.cc:1514] Processing DeleteTablet for tablet 1344bd342d224882a2b3babafbb1e995 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 06:30:15 UTC) from {username='kudu', principal='kudu/127.21.12.190@KRBTEST.COM'} at 127.0.0.1:35780
I20260806 06:30:15.506220 15198 tablet_replica.cc:330] T 1344bd342d224882a2b3babafbb1e995 P b267d2a459e34fefb8d9654d7b63ff45: stopping tablet replica
I20260806 06:30:15.506441 21554 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHhLhT/build/release/bin/kudu with pid 14473
I20260806 06:30:15.506381 15198 raft_consensus.cc:2229] T 1344bd342d224882a2b3babafbb1e995 P b267d2a459e34fefb8d9654d7b63ff45 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 06:30:15.510932 15197 tablet_replica.cc:330] T 1344bd342d224882a2b3babafbb1e995 P 911a0150a106481dbe2ce87e3c95f546: stopping tablet replica
I20260806 06:30:15.511114 15195 tablet_replica.cc:330] T 1344bd342d224882a2b3babafbb1e995 P 7d347dcbd0444c33a925274992622695: stopping tablet replica
I20260806 06:30:15.511667 15195 raft_consensus.cc:2229] T 1344bd342d224882a2b3babafbb1e995 P 7d347dcbd0444c33a925274992622695 [term 1 LEADER]: Raft consensus shutting down.
I20260806 06:30:15.511826 15197 raft_consensus.cc:2229] T 1344bd342d224882a2b3babafbb1e995 P 911a0150a106481dbe2ce87e3c95f546 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 06:30:15.511983 15195 raft_consensus.cc:2258] T 1344bd342d224882a2b3babafbb1e995 P 7d347dcbd0444c33a925274992622695 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 06:30:15.512154 15197 raft_consensus.cc:2258] T 1344bd342d224882a2b3babafbb1e995 P 911a0150a106481dbe2ce87e3c95f546 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 06:30:15.513768 15197 ts_tablet_manager.cc:1905] T 1344bd342d224882a2b3babafbb1e995 P 911a0150a106481dbe2ce87e3c95f546: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 06:30:15.520772 15195 ts_tablet_manager.cc:1905] T 1344bd342d224882a2b3babafbb1e995 P 7d347dcbd0444c33a925274992622695: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 06:30:15.522435 15197 ts_tablet_manager.cc:1918] T 1344bd342d224882a2b3babafbb1e995 P 911a0150a106481dbe2ce87e3c95f546: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 06:30:15.522521 15195 ts_tablet_manager.cc:1918] T 1344bd342d224882a2b3babafbb1e995 P 7d347dcbd0444c33a925274992622695: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 06:30:15.522574 15197 log.cc:1196] T 1344bd342d224882a2b3babafbb1e995 P 911a0150a106481dbe2ce87e3c95f546: Deleting WAL directory at /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785997581049705-21554-0/minicluster-data/ts-2/wal/wals/1344bd342d224882a2b3babafbb1e995
I20260806 06:30:15.522702 15195 log.cc:1196] T 1344bd342d224882a2b3babafbb1e995 P 7d347dcbd0444c33a925274992622695: Deleting WAL directory at /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785997581049705-21554-0/minicluster-data/ts-1/wal/wals/1344bd342d224882a2b3babafbb1e995
I20260806 06:30:15.522832 15197 ts_tablet_manager.cc:1939] T 1344bd342d224882a2b3babafbb1e995 P 911a0150a106481dbe2ce87e3c95f546: Deleting consensus metadata
I20260806 06:30:15.523140 15195 ts_tablet_manager.cc:1939] T 1344bd342d224882a2b3babafbb1e995 P 7d347dcbd0444c33a925274992622695: Deleting consensus metadata
I20260806 06:30:15.523425 14357 catalog_manager.cc:4841] TS 911a0150a106481dbe2ce87e3c95f546 (127.21.12.131:42901): tablet 1344bd342d224882a2b3babafbb1e995 (table test-table [id=57ecafca3b534c42aeb6bdf1a442c29f]) successfully deleted
I20260806 06:30:15.523761 14355 catalog_manager.cc:4841] TS 7d347dcbd0444c33a925274992622695 (127.21.12.130:40141): tablet 1344bd342d224882a2b3babafbb1e995 (table test-table [id=57ecafca3b534c42aeb6bdf1a442c29f]) successfully deleted
W20260806 06:30:15.524695 14355 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.12.129:37013 (error 108)
I20260806 06:30:15.524853 21554 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHhLhT/build/release/bin/kudu with pid 14694
I20260806 06:30:15.525673 15151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:15.525522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.129:37013 (local address 127.0.0.1:35796)
0806 06:30:15.525585 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0806 06:30:15.525633 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.129:37013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260806 06:30:15.525789 14355 catalog_manager.cc:4584] TS b267d2a459e34fefb8d9654d7b63ff45 (127.21.12.129:37013): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1344bd342d224882a2b3babafbb1e995: Network error: Client connection negotiation failed: client connection to 127.21.12.129:37013: connect: Connection refused (error 111)
I20260806 06:30:15.530509 21554 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHhLhT/build/release/bin/kudu with pid 14935
I20260806 06:30:15.539089 21554 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHhLhT/build/release/bin/kudu with pid 14330
2026-08-06T06:30:15Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2138 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-08-06T06:30:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T06:30:15Z Disabled control of system clock
I20260806 06:30:15.568289 21554 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHhLhT/build/release/bin/kudu
/tmp/dist-test-taskFHhLhT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785997581049705-21554-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785997581049705-21554-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785997581049705-21554-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785997581049705-21554-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.12.190:43557
--webserver_interface=127.21.12.190
--webserver_port=0
--builtin_ntp_servers=127.21.12.148:38255
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.12.190:43557
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260806 06:30:15.647563 15208 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 06:30:15.647765 15208 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 06:30:15.647784 15208 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 06:30:15.648912 15208 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 06:30:15.648953 15208 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 06:30:15.648965 15208 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 06:30:15.648979 15208 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 06:30:15.648993 15208 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 06:30:15.650400 15208 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.148:38255
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785997581049705-21554-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785997581049705-21554-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.12.190:43557
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.12.190:43557
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785997581049705-21554-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.12.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15208
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785997581049705-21554-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 05:11:02 UTC on ccycloud.quasar-hvzbbz.root.comops.site
build id 4569617
I20260806 06:30:15.650655 15208 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 06:30:15.650923 15208 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 06:30:15.655391 15220 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:30:15.655368 15217 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:30:15.655411 15218 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:30:15.655508 15208 server_base.cc:1034] running on GCE node
I20260806 06:30:15.655829 15208 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 06:30:15.656090 15208 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 06:30:15.657255 15208 hybrid_clock.cc:648] HybridClock initialized: now 1785997815657236 us; error 28 us; skew 500 ppm
I20260806 06:30:15.658449 15208 webserver.cc:460] Webserver started at http://127.21.12.190:33549/ using document root <none> and password file <none>
I20260806 06:30:15.658696 15208 fs_manager.cc:359] Metadata directory not provided
I20260806 06:30:15.658742 15208 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 06:30:15.658871 15208 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 06:30:15.659879 15208 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785997581049705-21554-0/minicluster-data/master-0/data/instance:
uuid: "98c3fef1e2c94297a766eb5fb4a60714"
format_stamp: "Formatted at 2026-08-06 06:30:15 on dist-test-slave-4tdj"
server_key: "6bd89fffcf5175b10af0b6a8cd193e4c"
server_key_iv: "cf4986736c4cc89d13c99f3b647cbf84"
server_key_version: "encryptionkey@0"
I20260806 06:30:15.660219 15208 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785997581049705-21554-0/minicluster-data/master-0/wal/instance:
uuid: "98c3fef1e2c94297a766eb5fb4a60714"
format_stamp: "Formatted at 2026-08-06 06:30:15 on dist-test-slave-4tdj"
server_key: "6bd89fffcf5175b10af0b6a8cd193e4c"
server_key_iv: "cf4986736c4cc89d13c99f3b647cbf84"
server_key_version: "encryptionkey@0"
I20260806 06:30:15.661420 15208 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260806 06:30:15.662078 15227 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 06:30:15.662213 15208 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260806 06:30:15.662282 15208 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785997581049705-21554-0/minicluster-data/master-0/data,/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785997581049705-21554-0/minicluster-data/master-0/wal
uuid: "98c3fef1e2c94297a766eb5fb4a60714"
format_stamp: "Formatted at 2026-08-06 06:30:15 on dist-test-slave-4tdj"
server_key: "6bd89fffcf5175b10af0b6a8cd193e4c"
server_key_iv: "cf4986736c4cc89d13c99f3b647cbf84"
server_key_version: "encryptionkey@0"
I20260806 06:30:15.662370 15208 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785997581049705-21554-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785997581049705-21554-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785997581049705-21554-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 06:30:15.678470 15208 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 06:30:15.678762 15208 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 06:30:15.678890 15208 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 06:30:15.682873 15208 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.190:43557
I20260806 06:30:15.682897 15307 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.190:43557 every 8 connection(s)
I20260806 06:30:15.683228 15208 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785997581049705-21554-0/minicluster-data/master-0/data/info.pb
I20260806 06:30:15.684144 15308 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 06:30:15.686451 15308 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98c3fef1e2c94297a766eb5fb4a60714: Bootstrap starting.
I20260806 06:30:15.687026 15308 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 98c3fef1e2c94297a766eb5fb4a60714: Neither blocks nor log segments found. Creating new log.
I20260806 06:30:15.687299 15308 log.cc:824] T 00000000000000000000000000000000 P 98c3fef1e2c94297a766eb5fb4a60714: Log is configured to *not* fsync() on all Append() calls
I20260806 06:30:15.687919 15308 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98c3fef1e2c94297a766eb5fb4a60714: No bootstrap required, opened a new log
I20260806 06:30:15.689498 15308 raft_consensus.cc:348] T 00000000000000000000000000000000 P 98c3fef1e2c94297a766eb5fb4a60714 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98c3fef1e2c94297a766eb5fb4a60714" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 43557 } }
I20260806 06:30:15.689635 15308 raft_consensus.cc:374] T 00000000000000000000000000000000 P 98c3fef1e2c94297a766eb5fb4a60714 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 06:30:15.689669 15308 raft_consensus.cc:729] T 00000000000000000000000000000000 P 98c3fef1e2c94297a766eb5fb4a60714 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98c3fef1e2c94297a766eb5fb4a60714, State: Initialized, Role: FOLLOWER
I20260806 06:30:15.689793 15308 consensus_queue.cc:260] T 00000000000000000000000000000000 P 98c3fef1e2c94297a766eb5fb4a60714 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98c3fef1e2c94297a766eb5fb4a60714" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 43557 } }
I20260806 06:30:15.689854 15308 raft_consensus.cc:388] T 00000000000000000000000000000000 P 98c3fef1e2c94297a766eb5fb4a60714 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 06:30:15.689885 15308 raft_consensus.cc:482] T 00000000000000000000000000000000 P 98c3fef1e2c94297a766eb5fb4a60714 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 06:30:15.689931 15308 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 98c3fef1e2c94297a766eb5fb4a60714 [term 0 FOLLOWER]: Advancing to term 1
I20260806 06:30:15.690466 15308 raft_consensus.cc:504] T 00000000000000000000000000000000 P 98c3fef1e2c94297a766eb5fb4a60714 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98c3fef1e2c94297a766eb5fb4a60714" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 43557 } }
I20260806 06:30:15.690562 15308 leader_election.cc:302] T 00000000000000000000000000000000 P 98c3fef1e2c94297a766eb5fb4a60714 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 98c3fef1e2c94297a766eb5fb4a60714; no voters:
I20260806 06:30:15.690716 15308 leader_election.cc:288] T 00000000000000000000000000000000 P 98c3fef1e2c94297a766eb5fb4a60714 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 06:30:15.690766 15312 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 98c3fef1e2c94297a766eb5fb4a60714 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 06:30:15.690951 15312 raft_consensus.cc:686] T 00000000000000000000000000000000 P 98c3fef1e2c94297a766eb5fb4a60714 [term 1 LEADER]: Becoming Leader. State: Replica: 98c3fef1e2c94297a766eb5fb4a60714, State: Running, Role: LEADER
I20260806 06:30:15.690953 15308 sys_catalog.cc:564] T 00000000000000000000000000000000 P 98c3fef1e2c94297a766eb5fb4a60714 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 06:30:15.691116 15312 consensus_queue.cc:237] T 00000000000000000000000000000000 P 98c3fef1e2c94297a766eb5fb4a60714 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98c3fef1e2c94297a766eb5fb4a60714" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 43557 } }
I20260806 06:30:15.691586 15314 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98c3fef1e2c94297a766eb5fb4a60714 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 98c3fef1e2c94297a766eb5fb4a60714. Latest consensus state: current_term: 1 leader_uuid: "98c3fef1e2c94297a766eb5fb4a60714" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98c3fef1e2c94297a766eb5fb4a60714" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 43557 } } }
I20260806 06:30:15.691671 15314 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98c3fef1e2c94297a766eb5fb4a60714 [sys.catalog]: This master's current role is: LEADER
I20260806 06:30:15.691743 15313 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98c3fef1e2c94297a766eb5fb4a60714 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "98c3fef1e2c94297a766eb5fb4a60714" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98c3fef1e2c94297a766eb5fb4a60714" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 43557 } } }
I20260806 06:30:15.692065 15313 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98c3fef1e2c94297a766eb5fb4a60714 [sys.catalog]: This master's current role is: LEADER
I20260806 06:30:15.692106 15322 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 06:30:15.692476 21554 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHhLhT/build/release/bin/kudu as pid 15208
I20260806 06:30:15.692593 21554 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785997581049705-21554-0/minicluster-data/master-0/wal/instance
I20260806 06:30:15.692613 15322 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 06:30:15.692818 21554 external_mini_cluster.cc:1428] Setting key 41f2b5d5e57b5f9b20da9c82e7331466
I20260806 06:30:15.694128 15322 catalog_manager.cc:1347] Generated new cluster ID: f52c74074d4743c9b0421ee0a5d382ca
I20260806 06:30:15.694181 15322 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 06:30:15.701905 15337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:15.693260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38474 (local address 127.21.12.190:43557)
0806 06:30:15.693447 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0806 06:30:15.693456 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 06:30:15.693529 (+ 73us) server_negotiation.cc:407] Connection header received
0806 06:30:15.693699 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:30:15.693706 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:30:15.693786 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:30:15.693925 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:30:15.698363 (+ 4438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:15.699247 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:15.700002 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:15.700490 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:15.700609 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:30:15.701229 (+ 620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:30:15.701233 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:30:15.701234 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 06:30:15.701401 (+ 167us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:30:15.701433 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 06:30:15.701444 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:30:15.701499 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0806 06:30:15.701608 (+ 109us) server_negotiation.cc:299] Negotiation successful
0806 06:30:15.701691 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":82,"threads_started":1}
I20260806 06:30:15.720212 15322 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 06:30:15.720746 15322 catalog_manager.cc:1504] Loading token signing keys...
I20260806 06:30:15.723441 15322 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 98c3fef1e2c94297a766eb5fb4a60714: Generated new TSK 0
I20260806 06:30:15.723615 15322 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 06:30:15.726200 21554 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHhLhT/build/release/bin/kudu
/tmp/dist-test-taskFHhLhT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785997581049705-21554-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785997581049705-21554-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785997581049705-21554-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785997581049705-21554-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.12.129:0
--local_ip_for_outbound_sockets=127.21.12.129
--webserver_interface=127.21.12.129
--webserver_port=0
--tserver_master_addrs=127.21.12.190:43557
--builtin_ntp_servers=127.21.12.148:38255
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260806 06:30:15.807197 15340 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 06:30:15.807404 15340 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 06:30:15.807436 15340 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 06:30:15.808573 15340 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 06:30:15.808612 15340 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 06:30:15.808643 15340 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.12.129
I20260806 06:30:15.809989 15340 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.148:38255
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785997581049705-21554-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785997581049705-21554-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.12.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785997581049705-21554-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.12.129
--webserver_port=0
--tserver_master_addrs=127.21.12.190:43557
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15340
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.12.129
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785997581049705-21554-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 05:11:02 UTC on ccycloud.quasar-hvzbbz.root.comops.site
build id 4569617
I20260806 06:30:15.810223 15340 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 06:30:15.810527 15340 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 06:30:15.816365 15346 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:30:15.816376 15349 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:30:15.816411 15351 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:30:15.816582 15340 server_base.cc:1034] running on GCE node
I20260806 06:30:15.816826 15340 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 06:30:15.817044 15340 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 06:30:15.818200 15340 hybrid_clock.cc:648] HybridClock initialized: now 1785997815818188 us; error 29 us; skew 500 ppm
I20260806 06:30:15.819392 15340 webserver.cc:460] Webserver started at http://127.21.12.129:40075/ using document root <none> and password file <none>
I20260806 06:30:15.819635 15340 fs_manager.cc:359] Metadata directory not provided
I20260806 06:30:15.819685 15340 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 06:30:15.819816 15340 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 06:30:15.820832 15340 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785997581049705-21554-0/minicluster-data/ts-0/data/instance:
uuid: "3f34b429ab2e401b9720ba371c22fe96"
format_stamp: "Formatted at 2026-08-06 06:30:15 on dist-test-slave-4tdj"
server_key: "e03b6b638eba9b5e02b3223365c333e1"
server_key_iv: "68220154966475ea53ed6a0b8b962387"
server_key_version: "encryptionkey@0"
I20260806 06:30:15.821164 15340 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785997581049705-21554-0/minicluster-data/ts-0/wal/instance:
uuid: "3f34b429ab2e401b9720ba371c22fe96"
format_stamp: "Formatted at 2026-08-06 06:30:15 on dist-test-slave-4tdj"
server_key: "e03b6b638eba9b5e02b3223365c333e1"
server_key_iv: "68220154966475ea53ed6a0b8b962387"
server_key_version: "encryptionkey@0"
I20260806 06:30:15.822427 15340 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260806 06:30:15.823175 15360 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 06:30:15.823333 15340 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260806 06:30:15.823418 15340 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785997581049705-21554-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785997581049705-21554-0/minicluster-data/ts-0/wal
uuid: "3f34b429ab2e401b9720ba371c22fe96"
format_stamp: "Formatted at 2026-08-06 06:30:15 on dist-test-slave-4tdj"
server_key: "e03b6b638eba9b5e02b3223365c333e1"
server_key_iv: "68220154966475ea53ed6a0b8b962387"
server_key_version: "encryptionkey@0"
I20260806 06:30:15.823485 15340 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785997581049705-21554-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785997581049705-21554-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785997581049705-21554-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 06:30:15.848608 15340 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 06:30:15.848930 15340 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 06:30:15.849063 15340 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 06:30:15.849299 15340 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 06:30:15.849627 15340 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 06:30:15.849663 15340 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 06:30:15.849694 15340 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 06:30:15.849712 15340 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 06:30:15.855338 15340 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.129:37779
I20260806 06:30:15.855391 15536 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.129:37779 every 8 connection(s)
I20260806 06:30:15.855705 15340 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785997581049705-21554-0/minicluster-data/ts-0/data/info.pb
I20260806 06:30:15.860702 15337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:15.856407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.129:33479 (local address 127.21.12.190:43557)
0806 06:30:15.856464 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0806 06:30:15.856466 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:30:15.856699 (+ 233us) server_negotiation.cc:407] Connection header received
0806 06:30:15.857206 (+ 507us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:30:15.857208 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:30:15.857223 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:30:15.857275 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:30:15.857928 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:15.858486 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:15.859418 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:15.859631 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:15.859686 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:30:15.860442 (+ 756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:30:15.860444 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:30:15.860444 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 06:30:15.860449 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:30:15.860464 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 06:30:15.860468 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:30:15.860513 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0806 06:30:15.860594 (+ 81us) server_negotiation.cc:299] Negotiation successful
0806 06:30:15.860624 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260806 06:30:15.860879 15544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:15.856385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:43557 (local address 127.21.12.129:33479)
0806 06:30:15.856584 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0806 06:30:15.856620 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0806 06:30:15.857110 (+ 490us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:30:15.857342 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:30:15.857346 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:30:15.857355 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0806 06:30:15.857847 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:30:15.857851 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:30:15.858568 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:30:15.858569 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:30:15.859338 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:30:15.859340 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:30:15.859702 (+ 362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:30:15.859703 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:30:15.859827 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:30:15.860312 (+ 485us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 06:30:15.860313 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0806 06:30:15.860341 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 06:30:15.860364 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 06:30:15.860521 (+ 157us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 06:30:15.860527 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 06:30:15.860530 (+ 3us) client_negotiation.cc:769] Sending connection context
0806 06:30:15.860588 (+ 58us) client_negotiation.cc:240] Negotiation successful
0806 06:30:15.860688 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":95,"threads_started":1}
I20260806 06:30:15.861294 15539 heartbeater.cc:347] Connected to a master server at 127.21.12.190:43557
I20260806 06:30:15.861374 15539 heartbeater.cc:464] Registering TS with master...
I20260806 06:30:15.861514 15539 heartbeater.cc:511] Master 127.21.12.190:43557 requested a full tablet report, sending...
I20260806 06:30:15.861953 15259 ts_manager.cc:194] Registered new tserver with Master: 3f34b429ab2e401b9720ba371c22fe96 (127.21.12.129:37779)
I20260806 06:30:15.861958 21554 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHhLhT/build/release/bin/kudu as pid 15340
I20260806 06:30:15.862046 21554 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785997581049705-21554-0/minicluster-data/ts-0/wal/instance
I20260806 06:30:15.862226 21554 external_mini_cluster.cc:1428] Setting key ca114149a490b174289908194fe919cb
I20260806 06:30:15.862757 15259 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.12.129:33479
I20260806 06:30:15.863278 21554 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHhLhT/build/release/bin/kudu
/tmp/dist-test-taskFHhLhT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785997581049705-21554-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785997581049705-21554-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785997581049705-21554-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785997581049705-21554-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.12.130:0
--local_ip_for_outbound_sockets=127.21.12.130
--webserver_interface=127.21.12.130
--webserver_port=0
--tserver_master_addrs=127.21.12.190:43557
--builtin_ntp_servers=127.21.12.148:38255
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260806 06:30:15.943173 15545 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 06:30:15.943372 15545 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 06:30:15.943403 15545 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 06:30:15.944504 15545 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 06:30:15.944548 15545 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 06:30:15.944581 15545 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.12.130
I20260806 06:30:15.945919 15545 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.148:38255
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785997581049705-21554-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785997581049705-21554-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.12.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785997581049705-21554-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.12.130
--webserver_port=0
--tserver_master_addrs=127.21.12.190:43557
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15545
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.12.130
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785997581049705-21554-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 05:11:02 UTC on ccycloud.quasar-hvzbbz.root.comops.site
build id 4569617
I20260806 06:30:15.946136 15545 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 06:30:15.946424 15545 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 06:30:15.951095 15554 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:30:15.951120 15553 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:30:15.951120 15556 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:30:15.951298 15545 server_base.cc:1034] running on GCE node
I20260806 06:30:15.951485 15545 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 06:30:15.951740 15545 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 06:30:15.952888 15545 hybrid_clock.cc:648] HybridClock initialized: now 1785997815952874 us; error 28 us; skew 500 ppm
I20260806 06:30:15.954035 15545 webserver.cc:460] Webserver started at http://127.21.12.130:39011/ using document root <none> and password file <none>
I20260806 06:30:15.954249 15545 fs_manager.cc:359] Metadata directory not provided
I20260806 06:30:15.954320 15545 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 06:30:15.954455 15545 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 06:30:15.955449 15545 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785997581049705-21554-0/minicluster-data/ts-1/data/instance:
uuid: "69e5b67c1b9445a3b1ee7b99c0672baf"
format_stamp: "Formatted at 2026-08-06 06:30:15 on dist-test-slave-4tdj"
server_key: "419da5e57e14cdb1b58af11005539d86"
server_key_iv: "ace4a45318702fd5a6f90b3d61f1ec73"
server_key_version: "encryptionkey@0"
I20260806 06:30:15.955792 15545 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785997581049705-21554-0/minicluster-data/ts-1/wal/instance:
uuid: "69e5b67c1b9445a3b1ee7b99c0672baf"
format_stamp: "Formatted at 2026-08-06 06:30:15 on dist-test-slave-4tdj"
server_key: "419da5e57e14cdb1b58af11005539d86"
server_key_iv: "ace4a45318702fd5a6f90b3d61f1ec73"
server_key_version: "encryptionkey@0"
I20260806 06:30:15.956990 15545 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260806 06:30:15.957679 15563 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 06:30:15.957837 15545 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260806 06:30:15.957921 15545 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785997581049705-21554-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785997581049705-21554-0/minicluster-data/ts-1/wal
uuid: "69e5b67c1b9445a3b1ee7b99c0672baf"
format_stamp: "Formatted at 2026-08-06 06:30:15 on dist-test-slave-4tdj"
server_key: "419da5e57e14cdb1b58af11005539d86"
server_key_iv: "ace4a45318702fd5a6f90b3d61f1ec73"
server_key_version: "encryptionkey@0"
I20260806 06:30:15.957988 15545 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785997581049705-21554-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785997581049705-21554-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785997581049705-21554-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 06:30:15.966044 15545 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 06:30:15.966286 15545 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 06:30:15.966441 15545 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 06:30:15.966635 15545 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 06:30:15.966917 15545 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 06:30:15.966949 15545 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 06:30:15.966975 15545 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 06:30:15.967001 15545 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 06:30:15.972250 15545 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.130:38995
I20260806 06:30:15.972307 15722 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.130:38995 every 8 connection(s)
I20260806 06:30:15.972584 15545 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785997581049705-21554-0/minicluster-data/ts-1/data/info.pb
I20260806 06:30:15.977069 15337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:15.973207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.130:43561 (local address 127.21.12.190:43557)
0806 06:30:15.973268 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0806 06:30:15.973271 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 06:30:15.973483 (+ 212us) server_negotiation.cc:407] Connection header received
0806 06:30:15.973927 (+ 444us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:30:15.973928 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:30:15.973936 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:30:15.973990 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:30:15.974635 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:15.975121 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:15.975981 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:15.976165 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:15.976208 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:30:15.976898 (+ 690us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:30:15.976898 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:30:15.976899 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 06:30:15.976902 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:30:15.976911 (+ 9us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 06:30:15.976914 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:30:15.976945 (+ 31us) server_negotiation.cc:1035] Waiting for connection context
0806 06:30:15.976995 (+ 50us) server_negotiation.cc:299] Negotiation successful
0806 06:30:15.977013 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260806 06:30:15.977263 15726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:15.973194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:43557 (local address 127.21.12.130:43561)
0806 06:30:15.973367 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0806 06:30:15.973406 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0806 06:30:15.973853 (+ 447us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:30:15.974040 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:30:15.974044 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:30:15.974051 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0806 06:30:15.974563 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:30:15.974569 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:30:15.975182 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:30:15.975183 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:30:15.975925 (+ 742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:30:15.975927 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:30:15.976217 (+ 290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:30:15.976218 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:30:15.976317 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:30:15.976788 (+ 471us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 06:30:15.976789 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0806 06:30:15.976816 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 06:30:15.976838 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 06:30:15.976947 (+ 109us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 06:30:15.976949 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 06:30:15.976951 (+ 2us) client_negotiation.cc:769] Sending connection context
0806 06:30:15.976994 (+ 43us) client_negotiation.cc:240] Negotiation successful
0806 06:30:15.977072 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":81,"threads_started":1}
I20260806 06:30:15.977799 15724 heartbeater.cc:347] Connected to a master server at 127.21.12.190:43557
I20260806 06:30:15.977885 15724 heartbeater.cc:464] Registering TS with master...
I20260806 06:30:15.978063 15724 heartbeater.cc:511] Master 127.21.12.190:43557 requested a full tablet report, sending...
I20260806 06:30:15.978134 21554 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHhLhT/build/release/bin/kudu as pid 15545
I20260806 06:30:15.978255 21554 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785997581049705-21554-0/minicluster-data/ts-1/wal/instance
I20260806 06:30:15.978502 15259 ts_manager.cc:194] Registered new tserver with Master: 69e5b67c1b9445a3b1ee7b99c0672baf (127.21.12.130:38995)
I20260806 06:30:15.978523 21554 external_mini_cluster.cc:1428] Setting key 6bb78fcf543ee79b9fa0db3a2f79b7ac
I20260806 06:30:15.978930 15259 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.12.130:43561
I20260806 06:30:15.979425 21554 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHhLhT/build/release/bin/kudu
/tmp/dist-test-taskFHhLhT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785997581049705-21554-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785997581049705-21554-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785997581049705-21554-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785997581049705-21554-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.12.131:0
--local_ip_for_outbound_sockets=127.21.12.131
--webserver_interface=127.21.12.131
--webserver_port=0
--tserver_master_addrs=127.21.12.190:43557
--builtin_ntp_servers=127.21.12.148:38255
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260806 06:30:16.059724 15727 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 06:30:16.059914 15727 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 06:30:16.059943 15727 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 06:30:16.061079 15727 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 06:30:16.061117 15727 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 06:30:16.061149 15727 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.12.131
I20260806 06:30:16.062518 15727 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.148:38255
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785997581049705-21554-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785997581049705-21554-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.12.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785997581049705-21554-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.12.131
--webserver_port=0
--tserver_master_addrs=127.21.12.190:43557
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15727
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.12.131
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785997581049705-21554-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 05:11:02 UTC on ccycloud.quasar-hvzbbz.root.comops.site
build id 4569617
I20260806 06:30:16.062737 15727 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 06:30:16.062994 15727 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 06:30:16.067689 15737 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:30:16.067708 15736 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:30:16.067734 15727 server_base.cc:1034] running on GCE node
W20260806 06:30:16.067723 15742 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:30:16.068121 15727 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 06:30:16.068328 15727 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 06:30:16.069476 15727 hybrid_clock.cc:648] HybridClock initialized: now 1785997816069462 us; error 29 us; skew 500 ppm
I20260806 06:30:16.075094 15727 webserver.cc:460] Webserver started at http://127.21.12.131:43091/ using document root <none> and password file <none>
I20260806 06:30:16.075305 15727 fs_manager.cc:359] Metadata directory not provided
I20260806 06:30:16.075354 15727 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 06:30:16.075480 15727 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 06:30:16.076490 15727 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785997581049705-21554-0/minicluster-data/ts-2/data/instance:
uuid: "3f846f1c3cb9406abff185e3dd441843"
format_stamp: "Formatted at 2026-08-06 06:30:16 on dist-test-slave-4tdj"
server_key: "09721eb3a65f813b2643d0540a59f5ac"
server_key_iv: "eeb73ac7597907f327f41541d616ca43"
server_key_version: "encryptionkey@0"
I20260806 06:30:16.076814 15727 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785997581049705-21554-0/minicluster-data/ts-2/wal/instance:
uuid: "3f846f1c3cb9406abff185e3dd441843"
format_stamp: "Formatted at 2026-08-06 06:30:16 on dist-test-slave-4tdj"
server_key: "09721eb3a65f813b2643d0540a59f5ac"
server_key_iv: "eeb73ac7597907f327f41541d616ca43"
server_key_version: "encryptionkey@0"
I20260806 06:30:16.078013 15727 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260806 06:30:16.078692 15752 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 06:30:16.078838 15727 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260806 06:30:16.078920 15727 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785997581049705-21554-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785997581049705-21554-0/minicluster-data/ts-2/wal
uuid: "3f846f1c3cb9406abff185e3dd441843"
format_stamp: "Formatted at 2026-08-06 06:30:16 on dist-test-slave-4tdj"
server_key: "09721eb3a65f813b2643d0540a59f5ac"
server_key_iv: "eeb73ac7597907f327f41541d616ca43"
server_key_version: "encryptionkey@0"
I20260806 06:30:16.078984 15727 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785997581049705-21554-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785997581049705-21554-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785997581049705-21554-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 06:30:16.085702 15727 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 06:30:16.085947 15727 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 06:30:16.086063 15727 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 06:30:16.086270 15727 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 06:30:16.086582 15727 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 06:30:16.086616 15727 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 06:30:16.086643 15727 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 06:30:16.086669 15727 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 06:30:16.092180 15727 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.131:39409
I20260806 06:30:16.092216 15927 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.131:39409 every 8 connection(s)
I20260806 06:30:16.092502 15727 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785997581049705-21554-0/minicluster-data/ts-2/data/info.pb
I20260806 06:30:16.094398 21554 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHhLhT/build/release/bin/kudu as pid 15727
I20260806 06:30:16.094486 21554 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785997581049705-21554-0/minicluster-data/ts-2/wal/instance
I20260806 06:30:16.094703 21554 external_mini_cluster.cc:1428] Setting key 235834998c75ab110c69fa7e2073df86
I20260806 06:30:16.097366 15337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:16.093144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.131:36151 (local address 127.21.12.190:43557)
0806 06:30:16.093206 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0806 06:30:16.093208 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:30:16.093405 (+ 197us) server_negotiation.cc:407] Connection header received
0806 06:30:16.093841 (+ 436us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:30:16.093842 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:30:16.093852 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:30:16.093896 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:30:16.094586 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:16.095296 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:16.096290 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:16.096474 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:16.096519 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:30:16.097174 (+ 655us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:30:16.097175 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:30:16.097175 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 06:30:16.097179 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:30:16.097193 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 06:30:16.097197 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:30:16.097233 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0806 06:30:16.097285 (+ 52us) server_negotiation.cc:299] Negotiation successful
0806 06:30:16.097309 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260806 06:30:16.097530 15931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:16.093130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:43557 (local address 127.21.12.131:36151)
0806 06:30:16.093296 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0806 06:30:16.093327 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0806 06:30:16.093771 (+ 444us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:30:16.093950 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:30:16.093953 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:30:16.093961 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0806 06:30:16.094503 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:30:16.094510 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:30:16.095387 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:30:16.095389 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:30:16.096219 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:30:16.096222 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:30:16.096526 (+ 304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:30:16.096527 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:30:16.096649 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:30:16.097056 (+ 407us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 06:30:16.097057 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0806 06:30:16.097087 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 06:30:16.097107 (+ 20us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 06:30:16.097235 (+ 128us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 06:30:16.097237 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 06:30:16.097239 (+ 2us) client_negotiation.cc:769] Sending connection context
0806 06:30:16.097281 (+ 42us) client_negotiation.cc:240] Negotiation successful
0806 06:30:16.097353 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":83,"threads_started":1}
I20260806 06:30:16.097954 15928 heartbeater.cc:347] Connected to a master server at 127.21.12.190:43557
I20260806 06:30:16.098035 15928 heartbeater.cc:464] Registering TS with master...
I20260806 06:30:16.098176 15928 heartbeater.cc:511] Master 127.21.12.190:43557 requested a full tablet report, sending...
I20260806 06:30:16.098493 15259 ts_manager.cc:194] Registered new tserver with Master: 3f846f1c3cb9406abff185e3dd441843 (127.21.12.131:39409)
I20260806 06:30:16.098814 15259 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.12.131:36151
I20260806 06:30:16.106355 21554 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 06:30:16.110206 15337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:16.107606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38480 (local address 127.21.12.190:43557)
0806 06:30:16.107664 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0806 06:30:16.107666 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:30:16.107757 (+ 91us) server_negotiation.cc:407] Connection header received
0806 06:30:16.107821 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:30:16.107822 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:30:16.107833 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:30:16.107879 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:30:16.108031 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:16.108481 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:16.109204 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:16.109388 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:16.109438 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:30:16.109981 (+ 543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:30:16.109982 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:30:16.109982 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 06:30:16.109987 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:30:16.110001 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 06:30:16.110005 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:30:16.110044 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0806 06:30:16.110112 (+ 68us) server_negotiation.cc:299] Negotiation successful
0806 06:30:16.110134 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260806 06:30:16.111469 15259 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38480:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 06:30:16.112936 15259 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 06:30:16.119176 15942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:16.114975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.130:38995 (local address 127.0.0.1:34308)
0806 06:30:16.115228 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0806 06:30:16.115272 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0806 06:30:16.115337 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:30:16.115693 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:30:16.115696 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:30:16.115698 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 06:30:16.115752 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:30:16.115753 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:30:16.117042 (+ 1289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:30:16.117043 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:30:16.118176 (+ 1133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:30:16.118178 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:30:16.118882 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:30:16.118883 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:30:16.118969 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:30:16.118986 (+ 17us) client_negotiation.cc:769] Sending connection context
0806 06:30:16.119039 (+ 53us) client_negotiation.cc:240] Negotiation successful
0806 06:30:16.119106 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":221,"mutex_wait_us":48,"thread_start_us":131,"threads_started":1}
I20260806 06:30:16.119297 15941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:16.114977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.131:39409 (local address 127.0.0.1:51536)
0806 06:30:16.115771 (+ 794us) negotiation.cc:107] Waiting for socket to connect
0806 06:30:16.115781 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0806 06:30:16.115827 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:30:16.115999 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:30:16.115999 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:30:16.116001 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 06:30:16.116029 (+ 28us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:30:16.116030 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:30:16.116796 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:30:16.116797 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:30:16.118066 (+ 1269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:30:16.118069 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:30:16.119057 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:30:16.119059 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:30:16.119128 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:30:16.119132 (+ 4us) client_negotiation.cc:769] Sending connection context
0806 06:30:16.119181 (+ 49us) client_negotiation.cc:240] Negotiation successful
0806 06:30:16.119224 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":779,"mutex_wait_us":2,"thread_start_us":50,"threads_started":1}
I20260806 06:30:16.119493 15945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:16.115060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34308 (local address 127.21.12.130:38995)
0806 06:30:16.115509 (+ 449us) server_negotiation.cc:206] Beginning negotiation
0806 06:30:16.115513 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 06:30:16.115524 (+ 11us) server_negotiation.cc:407] Connection header received
0806 06:30:16.115567 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:30:16.115569 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:30:16.115586 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:30:16.115675 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 06:30:16.115829 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:16.116940 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:16.118234 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:16.118790 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:16.119222 (+ 432us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:30:16.119323 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0806 06:30:16.119349 (+ 26us) server_negotiation.cc:299] Negotiation successful
0806 06:30:16.119410 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":411,"thread_start_us":71,"threads_started":1}
I20260806 06:30:16.119494 15946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:16.115229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60076 (local address 127.21.12.129:37779)
0806 06:30:16.115384 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0806 06:30:16.115389 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 06:30:16.115563 (+ 174us) server_negotiation.cc:407] Connection header received
0806 06:30:16.115639 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:30:16.115642 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:30:16.115657 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:30:16.115738 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 06:30:16.115950 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:16.117026 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:16.117949 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:16.118648 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:16.118722 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:30:16.118813 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0806 06:30:16.119381 (+ 568us) server_negotiation.cc:299] Negotiation successful
0806 06:30:16.119419 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":73,"threads_started":1}
I20260806 06:30:16.119176 15943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:16.114977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.129:37779 (local address 127.0.0.1:60076)
0806 06:30:16.115439 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0806 06:30:16.115447 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0806 06:30:16.115493 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:30:16.115814 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:30:16.115815 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:30:16.115817 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 06:30:16.115862 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:30:16.115863 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:30:16.117102 (+ 1239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:30:16.117102 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:30:16.117861 (+ 759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:30:16.117863 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:30:16.118733 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:30:16.118735 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:30:16.118969 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:30:16.118986 (+ 17us) client_negotiation.cc:769] Sending connection context
0806 06:30:16.119056 (+ 70us) client_negotiation.cc:240] Negotiation successful
0806 06:30:16.119095 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":446,"mutex_wait_us":5,"thread_start_us":119,"threads_started":1}
I20260806 06:30:16.119611 15949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:16.115219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51536 (local address 127.21.12.131:39409)
0806 06:30:16.115403 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0806 06:30:16.115407 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 06:30:16.115879 (+ 472us) server_negotiation.cc:407] Connection header received
0806 06:30:16.115920 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:30:16.115922 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:30:16.115938 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:30:16.116003 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 06:30:16.116075 (+ 72us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:16.116724 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:16.118161 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:16.118870 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:16.118932 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:30:16.119022 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0806 06:30:16.119209 (+ 187us) server_negotiation.cc:299] Negotiation successful
0806 06:30:16.119525 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":89,"threads_started":1}
I20260806 06:30:16.120239 15822 tablet_service.cc:1467] Processing CreateTablet for tablet b224d76730e54bd38f2f73d38877fad5 (DEFAULT_TABLE table=test-table [id=b982ad2b22e14f7488295d0bcb1cb0a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 06:30:16.120244 15430 tablet_service.cc:1467] Processing CreateTablet for tablet b224d76730e54bd38f2f73d38877fad5 (DEFAULT_TABLE table=test-table [id=b982ad2b22e14f7488295d0bcb1cb0a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 06:30:16.120244 15627 tablet_service.cc:1467] Processing CreateTablet for tablet b224d76730e54bd38f2f73d38877fad5 (DEFAULT_TABLE table=test-table [id=b982ad2b22e14f7488295d0bcb1cb0a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 06:30:16.120580 15822 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b224d76730e54bd38f2f73d38877fad5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 06:30:16.120584 15430 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b224d76730e54bd38f2f73d38877fad5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 06:30:16.120584 15627 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b224d76730e54bd38f2f73d38877fad5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 06:30:16.123171 15950 tablet_bootstrap.cc:492] T b224d76730e54bd38f2f73d38877fad5 P 3f846f1c3cb9406abff185e3dd441843: Bootstrap starting.
I20260806 06:30:16.123772 15951 tablet_bootstrap.cc:492] T b224d76730e54bd38f2f73d38877fad5 P 3f34b429ab2e401b9720ba371c22fe96: Bootstrap starting.
I20260806 06:30:16.123883 15950 tablet_bootstrap.cc:654] T b224d76730e54bd38f2f73d38877fad5 P 3f846f1c3cb9406abff185e3dd441843: Neither blocks nor log segments found. Creating new log.
I20260806 06:30:16.124046 15952 tablet_bootstrap.cc:492] T b224d76730e54bd38f2f73d38877fad5 P 69e5b67c1b9445a3b1ee7b99c0672baf: Bootstrap starting.
I20260806 06:30:16.124300 15950 log.cc:824] T b224d76730e54bd38f2f73d38877fad5 P 3f846f1c3cb9406abff185e3dd441843: Log is configured to *not* fsync() on all Append() calls
I20260806 06:30:16.124434 15951 tablet_bootstrap.cc:654] T b224d76730e54bd38f2f73d38877fad5 P 3f34b429ab2e401b9720ba371c22fe96: Neither blocks nor log segments found. Creating new log.
I20260806 06:30:16.124512 15952 tablet_bootstrap.cc:654] T b224d76730e54bd38f2f73d38877fad5 P 69e5b67c1b9445a3b1ee7b99c0672baf: Neither blocks nor log segments found. Creating new log.
I20260806 06:30:16.124748 15951 log.cc:824] T b224d76730e54bd38f2f73d38877fad5 P 3f34b429ab2e401b9720ba371c22fe96: Log is configured to *not* fsync() on all Append() calls
I20260806 06:30:16.124778 15952 log.cc:824] T b224d76730e54bd38f2f73d38877fad5 P 69e5b67c1b9445a3b1ee7b99c0672baf: Log is configured to *not* fsync() on all Append() calls
I20260806 06:30:16.125191 15950 tablet_bootstrap.cc:492] T b224d76730e54bd38f2f73d38877fad5 P 3f846f1c3cb9406abff185e3dd441843: No bootstrap required, opened a new log
I20260806 06:30:16.125276 15950 ts_tablet_manager.cc:1397] T b224d76730e54bd38f2f73d38877fad5 P 3f846f1c3cb9406abff185e3dd441843: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260806 06:30:16.125499 15952 tablet_bootstrap.cc:492] T b224d76730e54bd38f2f73d38877fad5 P 69e5b67c1b9445a3b1ee7b99c0672baf: No bootstrap required, opened a new log
I20260806 06:30:16.125566 15952 ts_tablet_manager.cc:1397] T b224d76730e54bd38f2f73d38877fad5 P 69e5b67c1b9445a3b1ee7b99c0672baf: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260806 06:30:16.126638 15951 tablet_bootstrap.cc:492] T b224d76730e54bd38f2f73d38877fad5 P 3f34b429ab2e401b9720ba371c22fe96: No bootstrap required, opened a new log
I20260806 06:30:16.126706 15951 ts_tablet_manager.cc:1397] T b224d76730e54bd38f2f73d38877fad5 P 3f34b429ab2e401b9720ba371c22fe96: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260806 06:30:16.127782 15950 raft_consensus.cc:348] T b224d76730e54bd38f2f73d38877fad5 P 3f846f1c3cb9406abff185e3dd441843 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69e5b67c1b9445a3b1ee7b99c0672baf" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 38995 } } peers { permanent_uuid: "3f846f1c3cb9406abff185e3dd441843" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 39409 } } peers { permanent_uuid: "3f34b429ab2e401b9720ba371c22fe96" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 37779 } }
I20260806 06:30:16.127954 15950 raft_consensus.cc:374] T b224d76730e54bd38f2f73d38877fad5 P 3f846f1c3cb9406abff185e3dd441843 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 06:30:16.127990 15950 raft_consensus.cc:729] T b224d76730e54bd38f2f73d38877fad5 P 3f846f1c3cb9406abff185e3dd441843 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f846f1c3cb9406abff185e3dd441843, State: Initialized, Role: FOLLOWER
I20260806 06:30:16.127925 15952 raft_consensus.cc:348] T b224d76730e54bd38f2f73d38877fad5 P 69e5b67c1b9445a3b1ee7b99c0672baf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69e5b67c1b9445a3b1ee7b99c0672baf" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 38995 } } peers { permanent_uuid: "3f846f1c3cb9406abff185e3dd441843" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 39409 } } peers { permanent_uuid: "3f34b429ab2e401b9720ba371c22fe96" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 37779 } }
I20260806 06:30:16.128053 15952 raft_consensus.cc:374] T b224d76730e54bd38f2f73d38877fad5 P 69e5b67c1b9445a3b1ee7b99c0672baf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 06:30:16.128103 15952 raft_consensus.cc:729] T b224d76730e54bd38f2f73d38877fad5 P 69e5b67c1b9445a3b1ee7b99c0672baf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69e5b67c1b9445a3b1ee7b99c0672baf, State: Initialized, Role: FOLLOWER
I20260806 06:30:16.128131 15950 consensus_queue.cc:260] T b224d76730e54bd38f2f73d38877fad5 P 3f846f1c3cb9406abff185e3dd441843 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69e5b67c1b9445a3b1ee7b99c0672baf" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 38995 } } peers { permanent_uuid: "3f846f1c3cb9406abff185e3dd441843" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 39409 } } peers { permanent_uuid: "3f34b429ab2e401b9720ba371c22fe96" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 37779 } }
I20260806 06:30:16.128203 15952 consensus_queue.cc:260] T b224d76730e54bd38f2f73d38877fad5 P 69e5b67c1b9445a3b1ee7b99c0672baf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69e5b67c1b9445a3b1ee7b99c0672baf" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 38995 } } peers { permanent_uuid: "3f846f1c3cb9406abff185e3dd441843" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 39409 } } peers { permanent_uuid: "3f34b429ab2e401b9720ba371c22fe96" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 37779 } }
I20260806 06:30:16.128396 15950 ts_tablet_manager.cc:1428] T b224d76730e54bd38f2f73d38877fad5 P 3f846f1c3cb9406abff185e3dd441843: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260806 06:30:16.128417 15952 ts_tablet_manager.cc:1428] T b224d76730e54bd38f2f73d38877fad5 P 69e5b67c1b9445a3b1ee7b99c0672baf: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260806 06:30:16.128530 15928 heartbeater.cc:503] Master 127.21.12.190:43557 was elected leader, sending a full tablet report...
I20260806 06:30:16.128633 15951 raft_consensus.cc:348] T b224d76730e54bd38f2f73d38877fad5 P 3f34b429ab2e401b9720ba371c22fe96 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69e5b67c1b9445a3b1ee7b99c0672baf" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 38995 } } peers { permanent_uuid: "3f846f1c3cb9406abff185e3dd441843" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 39409 } } peers { permanent_uuid: "3f34b429ab2e401b9720ba371c22fe96" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 37779 } }
I20260806 06:30:16.128777 15951 raft_consensus.cc:374] T b224d76730e54bd38f2f73d38877fad5 P 3f34b429ab2e401b9720ba371c22fe96 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 06:30:16.128819 15951 raft_consensus.cc:729] T b224d76730e54bd38f2f73d38877fad5 P 3f34b429ab2e401b9720ba371c22fe96 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f34b429ab2e401b9720ba371c22fe96, State: Initialized, Role: FOLLOWER
I20260806 06:30:16.128851 15724 heartbeater.cc:503] Master 127.21.12.190:43557 was elected leader, sending a full tablet report...
I20260806 06:30:16.128916 15951 consensus_queue.cc:260] T b224d76730e54bd38f2f73d38877fad5 P 3f34b429ab2e401b9720ba371c22fe96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69e5b67c1b9445a3b1ee7b99c0672baf" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 38995 } } peers { permanent_uuid: "3f846f1c3cb9406abff185e3dd441843" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 39409 } } peers { permanent_uuid: "3f34b429ab2e401b9720ba371c22fe96" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 37779 } }
I20260806 06:30:16.129175 15951 ts_tablet_manager.cc:1428] T b224d76730e54bd38f2f73d38877fad5 P 3f34b429ab2e401b9720ba371c22fe96: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260806 06:30:16.129199 15539 heartbeater.cc:503] Master 127.21.12.190:43557 was elected leader, sending a full tablet report...
W20260806 06:30:16.223093 15725 tablet.cc:2369] T b224d76730e54bd38f2f73d38877fad5 P 69e5b67c1b9445a3b1ee7b99c0672baf: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 06:30:16.343050 15930 tablet.cc:2369] T b224d76730e54bd38f2f73d38877fad5 P 3f846f1c3cb9406abff185e3dd441843: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 06:30:16.356356 15541 tablet.cc:2369] T b224d76730e54bd38f2f73d38877fad5 P 3f34b429ab2e401b9720ba371c22fe96: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 06:30:16.454986 15962 raft_consensus.cc:482] T b224d76730e54bd38f2f73d38877fad5 P 3f34b429ab2e401b9720ba371c22fe96 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 06:30:16.455130 15962 raft_consensus.cc:504] T b224d76730e54bd38f2f73d38877fad5 P 3f34b429ab2e401b9720ba371c22fe96 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69e5b67c1b9445a3b1ee7b99c0672baf" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 38995 } } peers { permanent_uuid: "3f846f1c3cb9406abff185e3dd441843" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 39409 } } peers { permanent_uuid: "3f34b429ab2e401b9720ba371c22fe96" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 37779 } }
I20260806 06:30:16.455498 15962 leader_election.cc:288] T b224d76730e54bd38f2f73d38877fad5 P 3f34b429ab2e401b9720ba371c22fe96 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 69e5b67c1b9445a3b1ee7b99c0672baf (127.21.12.130:38995), 3f846f1c3cb9406abff185e3dd441843 (127.21.12.131:39409)
I20260806 06:30:16.458211 15963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:16.455658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.130:38995 (local address 127.21.12.129:45151)
0806 06:30:16.455779 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0806 06:30:16.455788 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0806 06:30:16.455870 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:30:16.456028 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:30:16.456028 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:30:16.456030 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 06:30:16.456099 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:30:16.456101 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:30:16.456672 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:30:16.456673 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:30:16.457489 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:30:16.457491 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:30:16.457990 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:30:16.457992 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:30:16.458059 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:30:16.458063 (+ 4us) client_negotiation.cc:769] Sending connection context
0806 06:30:16.458111 (+ 48us) client_negotiation.cc:240] Negotiation successful
0806 06:30:16.458140 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"mutex_wait_us":45,"thread_start_us":59,"threads_started":1}
I20260806 06:30:16.458246 15945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:16.455716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.129:45151 (local address 127.21.12.130:38995)
0806 06:30:16.455765 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0806 06:30:16.455767 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:30:16.455838 (+ 71us) server_negotiation.cc:407] Connection header received
0806 06:30:16.455952 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:30:16.455953 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:30:16.455962 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:30:16.456015 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 06:30:16.456164 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:16.456601 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:16.457574 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:16.457920 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:16.457974 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:30:16.457981 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0806 06:30:16.458139 (+ 158us) server_negotiation.cc:299] Negotiation successful
0806 06:30:16.458180 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260806 06:30:16.458351 15964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:16.455679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.131:39409 (local address 127.21.12.129:53907)
0806 06:30:16.455785 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0806 06:30:16.455796 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0806 06:30:16.455852 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:30:16.455988 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:30:16.455988 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:30:16.455990 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 06:30:16.456073 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:30:16.456075 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:30:16.456684 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:30:16.456685 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:30:16.457486 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:30:16.457488 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:30:16.458073 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:30:16.458074 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:30:16.458153 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:30:16.458157 (+ 4us) client_negotiation.cc:769] Sending connection context
0806 06:30:16.458218 (+ 61us) client_negotiation.cc:240] Negotiation successful
0806 06:30:16.458254 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":90,"mutex_wait_us":11,"thread_start_us":56,"threads_started":1}
I20260806 06:30:16.458346 15949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:16.455805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.129:53907 (local address 127.21.12.131:39409)
0806 06:30:16.455869 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0806 06:30:16.455871 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:30:16.455880 (+ 9us) server_negotiation.cc:407] Connection header received
0806 06:30:16.455912 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:30:16.455913 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:30:16.455924 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:30:16.455975 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 06:30:16.456130 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:16.456607 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:16.457567 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:16.457974 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:16.458044 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:30:16.458053 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0806 06:30:16.458226 (+ 173us) server_negotiation.cc:299] Negotiation successful
0806 06:30:16.458265 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260806 06:30:16.458647 15649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b224d76730e54bd38f2f73d38877fad5" candidate_uuid: "3f34b429ab2e401b9720ba371c22fe96" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69e5b67c1b9445a3b1ee7b99c0672baf" is_pre_election: true
I20260806 06:30:16.458675 15857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b224d76730e54bd38f2f73d38877fad5" candidate_uuid: "3f34b429ab2e401b9720ba371c22fe96" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f846f1c3cb9406abff185e3dd441843" is_pre_election: true
I20260806 06:30:16.458804 15649 raft_consensus.cc:2454] T b224d76730e54bd38f2f73d38877fad5 P 69e5b67c1b9445a3b1ee7b99c0672baf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f34b429ab2e401b9720ba371c22fe96 in term 0.
I20260806 06:30:16.458822 15857 raft_consensus.cc:2454] T b224d76730e54bd38f2f73d38877fad5 P 3f846f1c3cb9406abff185e3dd441843 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f34b429ab2e401b9720ba371c22fe96 in term 0.
I20260806 06:30:16.459007 15367 leader_election.cc:302] T b224d76730e54bd38f2f73d38877fad5 P 3f34b429ab2e401b9720ba371c22fe96 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f34b429ab2e401b9720ba371c22fe96, 69e5b67c1b9445a3b1ee7b99c0672baf; no voters:
I20260806 06:30:16.459146 15962 raft_consensus.cc:2781] T b224d76730e54bd38f2f73d38877fad5 P 3f34b429ab2e401b9720ba371c22fe96 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 06:30:16.459187 15962 raft_consensus.cc:482] T b224d76730e54bd38f2f73d38877fad5 P 3f34b429ab2e401b9720ba371c22fe96 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 06:30:16.459209 15962 raft_consensus.cc:3037] T b224d76730e54bd38f2f73d38877fad5 P 3f34b429ab2e401b9720ba371c22fe96 [term 0 FOLLOWER]: Advancing to term 1
I20260806 06:30:16.459905 15962 raft_consensus.cc:504] T b224d76730e54bd38f2f73d38877fad5 P 3f34b429ab2e401b9720ba371c22fe96 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69e5b67c1b9445a3b1ee7b99c0672baf" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 38995 } } peers { permanent_uuid: "3f846f1c3cb9406abff185e3dd441843" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 39409 } } peers { permanent_uuid: "3f34b429ab2e401b9720ba371c22fe96" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 37779 } }
I20260806 06:30:16.460026 15962 leader_election.cc:288] T b224d76730e54bd38f2f73d38877fad5 P 3f34b429ab2e401b9720ba371c22fe96 [CANDIDATE]: Term 1 election: Requested vote from peers 69e5b67c1b9445a3b1ee7b99c0672baf (127.21.12.130:38995), 3f846f1c3cb9406abff185e3dd441843 (127.21.12.131:39409)
I20260806 06:30:16.460207 15649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b224d76730e54bd38f2f73d38877fad5" candidate_uuid: "3f34b429ab2e401b9720ba371c22fe96" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69e5b67c1b9445a3b1ee7b99c0672baf"
I20260806 06:30:16.460243 15857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b224d76730e54bd38f2f73d38877fad5" candidate_uuid: "3f34b429ab2e401b9720ba371c22fe96" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f846f1c3cb9406abff185e3dd441843"
I20260806 06:30:16.460289 15649 raft_consensus.cc:3037] T b224d76730e54bd38f2f73d38877fad5 P 69e5b67c1b9445a3b1ee7b99c0672baf [term 0 FOLLOWER]: Advancing to term 1
I20260806 06:30:16.460304 15857 raft_consensus.cc:3037] T b224d76730e54bd38f2f73d38877fad5 P 3f846f1c3cb9406abff185e3dd441843 [term 0 FOLLOWER]: Advancing to term 1
I20260806 06:30:16.461149 15649 raft_consensus.cc:2454] T b224d76730e54bd38f2f73d38877fad5 P 69e5b67c1b9445a3b1ee7b99c0672baf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f34b429ab2e401b9720ba371c22fe96 in term 1.
I20260806 06:30:16.461133 15857 raft_consensus.cc:2454] T b224d76730e54bd38f2f73d38877fad5 P 3f846f1c3cb9406abff185e3dd441843 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f34b429ab2e401b9720ba371c22fe96 in term 1.
I20260806 06:30:16.461297 15367 leader_election.cc:302] T b224d76730e54bd38f2f73d38877fad5 P 3f34b429ab2e401b9720ba371c22fe96 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f34b429ab2e401b9720ba371c22fe96, 69e5b67c1b9445a3b1ee7b99c0672baf; no voters:
I20260806 06:30:16.461407 15962 raft_consensus.cc:2781] T b224d76730e54bd38f2f73d38877fad5 P 3f34b429ab2e401b9720ba371c22fe96 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 06:30:16.461551 15962 raft_consensus.cc:686] T b224d76730e54bd38f2f73d38877fad5 P 3f34b429ab2e401b9720ba371c22fe96 [term 1 LEADER]: Becoming Leader. State: Replica: 3f34b429ab2e401b9720ba371c22fe96, State: Running, Role: LEADER
I20260806 06:30:16.461643 15962 consensus_queue.cc:237] T b224d76730e54bd38f2f73d38877fad5 P 3f34b429ab2e401b9720ba371c22fe96 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69e5b67c1b9445a3b1ee7b99c0672baf" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 38995 } } peers { permanent_uuid: "3f846f1c3cb9406abff185e3dd441843" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 39409 } } peers { permanent_uuid: "3f34b429ab2e401b9720ba371c22fe96" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 37779 } }
I20260806 06:30:16.462466 15258 catalog_manager.cc:5495] T b224d76730e54bd38f2f73d38877fad5 P 3f34b429ab2e401b9720ba371c22fe96 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3f34b429ab2e401b9720ba371c22fe96 (127.21.12.129). New cstate: current_term: 1 leader_uuid: "3f34b429ab2e401b9720ba371c22fe96" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69e5b67c1b9445a3b1ee7b99c0672baf" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 38995 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3f846f1c3cb9406abff185e3dd441843" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 39409 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3f34b429ab2e401b9720ba371c22fe96" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 37779 } health_report { overall_health: HEALTHY } } }
I20260806 06:30:16.472658 15946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:16.470695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60078 (local address 127.21.12.129:37779)
0806 06:30:16.470735 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0806 06:30:16.470736 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0806 06:30:16.470745 (+ 9us) server_negotiation.cc:407] Connection header received
0806 06:30:16.470809 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:30:16.470810 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:30:16.470826 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:30:16.470879 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 06:30:16.470966 (+ 87us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:16.471398 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:16.472053 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:16.472223 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:16.472262 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:30:16.472357 (+ 95us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 06:30:16.472443 (+ 86us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 06:30:16.472496 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0806 06:30:16.472595 (+ 99us) server_negotiation.cc:299] Negotiation successful
0806 06:30:16.472610 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260806 06:30:16.474200 15857 raft_consensus.cc:1261] T b224d76730e54bd38f2f73d38877fad5 P 3f846f1c3cb9406abff185e3dd441843 [term 1 FOLLOWER]: Refusing update from remote peer 3f34b429ab2e401b9720ba371c22fe96: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 06:30:16.474215 15649 raft_consensus.cc:1261] T b224d76730e54bd38f2f73d38877fad5 P 69e5b67c1b9445a3b1ee7b99c0672baf [term 1 FOLLOWER]: Refusing update from remote peer 3f34b429ab2e401b9720ba371c22fe96: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 06:30:16.474555 15966 consensus_queue.cc:1035] T b224d76730e54bd38f2f73d38877fad5 P 3f34b429ab2e401b9720ba371c22fe96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f846f1c3cb9406abff185e3dd441843" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 39409 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 06:30:16.474679 15962 consensus_queue.cc:1035] T b224d76730e54bd38f2f73d38877fad5 P 3f34b429ab2e401b9720ba371c22fe96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69e5b67c1b9445a3b1ee7b99c0672baf" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 38995 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 06:30:16.477178 15971 mvcc.cc:205] Tried to move back new op lower bound from 7315447056276144128 to 7315447056227454976. Current Snapshot: MvccSnapshot[applied={T|T < 7315447056276144128}]
I20260806 06:30:16.477485 15972 mvcc.cc:205] Tried to move back new op lower bound from 7315447056276144128 to 7315447056227454976. Current Snapshot: MvccSnapshot[applied={T|T < 7315447056276144128}]
I20260806 06:30:16.479146 15259 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:38480:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 06:30:16.479241 15259 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:38480:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 06:30:16.479926 15259 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 98c3fef1e2c94297a766eb5fb4a60714: Sending DeleteTablet for 3 replicas of tablet b224d76730e54bd38f2f73d38877fad5
I20260806 06:30:16.480264 15822 tablet_service.cc:1514] Processing DeleteTablet for tablet b224d76730e54bd38f2f73d38877fad5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 06:30:16 UTC) from {username='slave'} at 127.0.0.1:51536
I20260806 06:30:16.480317 15430 tablet_service.cc:1514] Processing DeleteTablet for tablet b224d76730e54bd38f2f73d38877fad5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 06:30:16 UTC) from {username='slave'} at 127.0.0.1:60076
I20260806 06:30:16.480553 15980 tablet_replica.cc:330] T b224d76730e54bd38f2f73d38877fad5 P 3f846f1c3cb9406abff185e3dd441843: stopping tablet replica
I20260806 06:30:16.480533 15627 tablet_service.cc:1514] Processing DeleteTablet for tablet b224d76730e54bd38f2f73d38877fad5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 06:30:16 UTC) from {username='slave'} at 127.0.0.1:34308
I20260806 06:30:16.480647 15980 raft_consensus.cc:2229] T b224d76730e54bd38f2f73d38877fad5 P 3f846f1c3cb9406abff185e3dd441843 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 06:30:16.480737 15980 raft_consensus.cc:2258] T b224d76730e54bd38f2f73d38877fad5 P 3f846f1c3cb9406abff185e3dd441843 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 06:30:16.480800 21554 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHhLhT/build/release/bin/kudu with pid 15340
I20260806 06:30:16.481180 15980 ts_tablet_manager.cc:1905] T b224d76730e54bd38f2f73d38877fad5 P 3f846f1c3cb9406abff185e3dd441843: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 06:30:16.480901 15981 tablet_replica.cc:330] T b224d76730e54bd38f2f73d38877fad5 P 3f34b429ab2e401b9720ba371c22fe96: stopping tablet replica
I20260806 06:30:16.482771 15980 ts_tablet_manager.cc:1918] T b224d76730e54bd38f2f73d38877fad5 P 3f846f1c3cb9406abff185e3dd441843: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 06:30:16.482831 15980 log.cc:1196] T b224d76730e54bd38f2f73d38877fad5 P 3f846f1c3cb9406abff185e3dd441843: Deleting WAL directory at /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785997581049705-21554-0/minicluster-data/ts-2/wal/wals/b224d76730e54bd38f2f73d38877fad5
I20260806 06:30:16.483024 15980 ts_tablet_manager.cc:1939] T b224d76730e54bd38f2f73d38877fad5 P 3f846f1c3cb9406abff185e3dd441843: Deleting consensus metadata
I20260806 06:30:16.483331 15233 catalog_manager.cc:4841] TS 3f846f1c3cb9406abff185e3dd441843 (127.21.12.131:39409): tablet b224d76730e54bd38f2f73d38877fad5 (table test-table [id=b982ad2b22e14f7488295d0bcb1cb0a7]) successfully deleted
I20260806 06:30:16.489197 15982 tablet_replica.cc:330] T b224d76730e54bd38f2f73d38877fad5 P 69e5b67c1b9445a3b1ee7b99c0672baf: stopping tablet replica
I20260806 06:30:16.489308 15982 raft_consensus.cc:2229] T b224d76730e54bd38f2f73d38877fad5 P 69e5b67c1b9445a3b1ee7b99c0672baf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 06:30:16.489409 15982 raft_consensus.cc:2258] T b224d76730e54bd38f2f73d38877fad5 P 69e5b67c1b9445a3b1ee7b99c0672baf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 06:30:16.490453 15982 ts_tablet_manager.cc:1905] T b224d76730e54bd38f2f73d38877fad5 P 69e5b67c1b9445a3b1ee7b99c0672baf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 06:30:16.492174 15982 ts_tablet_manager.cc:1918] T b224d76730e54bd38f2f73d38877fad5 P 69e5b67c1b9445a3b1ee7b99c0672baf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 06:30:16.492282 15982 log.cc:1196] T b224d76730e54bd38f2f73d38877fad5 P 69e5b67c1b9445a3b1ee7b99c0672baf: Deleting WAL directory at /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785997581049705-21554-0/minicluster-data/ts-1/wal/wals/b224d76730e54bd38f2f73d38877fad5
I20260806 06:30:16.492545 15982 ts_tablet_manager.cc:1939] T b224d76730e54bd38f2f73d38877fad5 P 69e5b67c1b9445a3b1ee7b99c0672baf: Deleting consensus metadata
I20260806 06:30:16.492951 15235 catalog_manager.cc:4841] TS 69e5b67c1b9445a3b1ee7b99c0672baf (127.21.12.130:38995): tablet b224d76730e54bd38f2f73d38877fad5 (table test-table [id=b982ad2b22e14f7488295d0bcb1cb0a7]) successfully deleted
W20260806 06:30:16.494284 15231 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.12.129:37779 (error 108)
I20260806 06:30:16.494498 21554 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHhLhT/build/release/bin/kudu with pid 15545
I20260806 06:30:16.497313 15943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:16.494662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.129:37779 (local address 127.0.0.1:60086)
0806 06:30:16.497237 (+ 2575us) negotiation.cc:107] Waiting for socket to connect
0806 06:30:16.497284 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.129:37779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2554}
W20260806 06:30:16.497439 15231 catalog_manager.cc:4584] TS 3f34b429ab2e401b9720ba371c22fe96 (127.21.12.129:37779): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b224d76730e54bd38f2f73d38877fad5: Network error: Client connection negotiation failed: client connection to 127.21.12.129:37779: connect: Connection refused (error 111)
I20260806 06:30:16.501813 21554 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHhLhT/build/release/bin/kudu with pid 15727
I20260806 06:30:16.510738 21554 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHhLhT/build/release/bin/kudu with pid 15208
2026-08-06T06:30:16Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (969 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785997581049705-21554-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 06:30:16 dist-test-slave-4tdj krb5kdc[15988](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 06:30:16 dist-test-slave-4tdj krb5kdc[15988](info): set up 2 sockets
Aug 06 06:30:16 dist-test-slave-4tdj krb5kdc[15988](info): commencing operation
krb5kdc: starting...
W20260806 06:30:17.618994 21554 mini_kdc.cc:121] Time spent starting KDC: real 1.088s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785997581049705-21554-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785997581049705-21554-0/krb5kdc/test-user.keytab.
Aug 06 06:30:17 dist-test-slave-4tdj krb5kdc[15988](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997817, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T06:30:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T06:30:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.12.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.12.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.12.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.190.keytab.
Entry for principal kudu/127.21.12.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.190.keytab.
I20260806 06:30:17.732334 21554 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHhLhT/build/release/bin/kudu
/tmp/dist-test-taskFHhLhT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785997581049705-21554-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785997581049705-21554-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785997581049705-21554-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785997581049705-21554-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.12.190:41187
--webserver_interface=127.21.12.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.190.keytab
--principal=kudu/127.21.12.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.148:41159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.12.190:41187
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785997581049705-21554-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785997581049705-21554-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785997581049705-21554-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 06:30:17.812213 16003 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 06:30:17.812422 16003 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 06:30:17.812455 16003 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 06:30:17.813588 16003 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 06:30:17.813629 16003 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 06:30:17.813647 16003 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 06:30:17.813663 16003 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 06:30:17.813678 16003 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 06:30:17.815189 16003 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.148:41159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785997581049705-21554-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785997581049705-21554-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.12.190:41187
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.190.keytab
--principal=kudu/127.21.12.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.12.190:41187
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785997581049705-21554-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.12.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16003
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785997581049705-21554-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 05:11:02 UTC on ccycloud.quasar-hvzbbz.root.comops.site
build id 4569617
I20260806 06:30:17.815435 16003 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 06:30:17.815641 16003 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 06:30:17.820237 16010 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:30:17.820228 16013 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:30:17.820230 16018 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:30:17.820492 16003 server_base.cc:1034] running on GCE node
I20260806 06:30:17.820722 16003 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 06:30:17.820976 16003 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 06:30:17.822130 16003 hybrid_clock.cc:648] HybridClock initialized: now 1785997817822114 us; error 28 us; skew 500 ppm
Aug 06 06:30:17 dist-test-slave-4tdj krb5kdc[15988](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997817, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 06:30:17.824535 16003 init.cc:377] Logged in from keytab as kudu/127.21.12.190@KRBTEST.COM (short username kudu)
I20260806 06:30:17.824963 16003 webserver.cc:460] Webserver started at http://127.21.12.190:36327/ using document root <none> and password file <none>
I20260806 06:30:17.825156 16003 fs_manager.cc:359] Metadata directory not provided
I20260806 06:30:17.825196 16003 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 06:30:17.825294 16003 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 06:30:17.826226 16003 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785997581049705-21554-0/minicluster-data/master-0/data/instance:
uuid: "bbb2a969c88948d2848d8e7758b933c5"
format_stamp: "Formatted at 2026-08-06 06:30:17 on dist-test-slave-4tdj"
server_key: "7c13cc3b4fc3293bf863a4b12915bf0e"
server_key_iv: "6bae0cdaadea8016ea9d9795ea1bf63c"
server_key_version: "encryptionkey@0"
I20260806 06:30:17.826596 16003 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785997581049705-21554-0/minicluster-data/master-0/wal/instance:
uuid: "bbb2a969c88948d2848d8e7758b933c5"
format_stamp: "Formatted at 2026-08-06 06:30:17 on dist-test-slave-4tdj"
server_key: "7c13cc3b4fc3293bf863a4b12915bf0e"
server_key_iv: "6bae0cdaadea8016ea9d9795ea1bf63c"
server_key_version: "encryptionkey@0"
I20260806 06:30:17.827785 16003 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260806 06:30:17.828481 16028 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 06:30:17.828676 16003 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260806 06:30:17.828770 16003 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785997581049705-21554-0/minicluster-data/master-0/data,/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785997581049705-21554-0/minicluster-data/master-0/wal
uuid: "bbb2a969c88948d2848d8e7758b933c5"
format_stamp: "Formatted at 2026-08-06 06:30:17 on dist-test-slave-4tdj"
server_key: "7c13cc3b4fc3293bf863a4b12915bf0e"
server_key_iv: "6bae0cdaadea8016ea9d9795ea1bf63c"
server_key_version: "encryptionkey@0"
I20260806 06:30:17.828830 16003 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785997581049705-21554-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785997581049705-21554-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785997581049705-21554-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 06:30:17.840735 16003 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 06:30:17.842418 16003 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 06:30:17.842536 16003 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 06:30:17.846457 16003 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.190:41187
I20260806 06:30:17.846503 16108 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.190:41187 every 8 connection(s)
I20260806 06:30:17.846814 16003 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785997581049705-21554-0/minicluster-data/master-0/data/info.pb
I20260806 06:30:17.847681 16109 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 06:30:17.849843 16109 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bbb2a969c88948d2848d8e7758b933c5: Bootstrap starting.
I20260806 06:30:17.850409 16109 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bbb2a969c88948d2848d8e7758b933c5: Neither blocks nor log segments found. Creating new log.
I20260806 06:30:17.850677 16109 log.cc:824] T 00000000000000000000000000000000 P bbb2a969c88948d2848d8e7758b933c5: Log is configured to *not* fsync() on all Append() calls
I20260806 06:30:17.851366 16109 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bbb2a969c88948d2848d8e7758b933c5: No bootstrap required, opened a new log
I20260806 06:30:17.852927 16109 raft_consensus.cc:348] T 00000000000000000000000000000000 P bbb2a969c88948d2848d8e7758b933c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbb2a969c88948d2848d8e7758b933c5" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 41187 } }
I20260806 06:30:17.853060 16109 raft_consensus.cc:374] T 00000000000000000000000000000000 P bbb2a969c88948d2848d8e7758b933c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 06:30:17.853091 16109 raft_consensus.cc:729] T 00000000000000000000000000000000 P bbb2a969c88948d2848d8e7758b933c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbb2a969c88948d2848d8e7758b933c5, State: Initialized, Role: FOLLOWER
I20260806 06:30:17.853214 16109 consensus_queue.cc:260] T 00000000000000000000000000000000 P bbb2a969c88948d2848d8e7758b933c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbb2a969c88948d2848d8e7758b933c5" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 41187 } }
I20260806 06:30:17.853274 16109 raft_consensus.cc:388] T 00000000000000000000000000000000 P bbb2a969c88948d2848d8e7758b933c5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 06:30:17.853303 16109 raft_consensus.cc:482] T 00000000000000000000000000000000 P bbb2a969c88948d2848d8e7758b933c5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 06:30:17.853350 16109 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bbb2a969c88948d2848d8e7758b933c5 [term 0 FOLLOWER]: Advancing to term 1
I20260806 06:30:17.853904 16109 raft_consensus.cc:504] T 00000000000000000000000000000000 P bbb2a969c88948d2848d8e7758b933c5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbb2a969c88948d2848d8e7758b933c5" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 41187 } }
I20260806 06:30:17.854000 16109 leader_election.cc:302] T 00000000000000000000000000000000 P bbb2a969c88948d2848d8e7758b933c5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bbb2a969c88948d2848d8e7758b933c5; no voters:
I20260806 06:30:17.854151 16109 leader_election.cc:288] T 00000000000000000000000000000000 P bbb2a969c88948d2848d8e7758b933c5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 06:30:17.854197 16113 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bbb2a969c88948d2848d8e7758b933c5 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 06:30:17.854414 16109 sys_catalog.cc:564] T 00000000000000000000000000000000 P bbb2a969c88948d2848d8e7758b933c5 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 06:30:17.854399 16113 raft_consensus.cc:686] T 00000000000000000000000000000000 P bbb2a969c88948d2848d8e7758b933c5 [term 1 LEADER]: Becoming Leader. State: Replica: bbb2a969c88948d2848d8e7758b933c5, State: Running, Role: LEADER
I20260806 06:30:17.854571 16113 consensus_queue.cc:237] T 00000000000000000000000000000000 P bbb2a969c88948d2848d8e7758b933c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbb2a969c88948d2848d8e7758b933c5" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 41187 } }
I20260806 06:30:17.855033 16114 sys_catalog.cc:455] T 00000000000000000000000000000000 P bbb2a969c88948d2848d8e7758b933c5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bbb2a969c88948d2848d8e7758b933c5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbb2a969c88948d2848d8e7758b933c5" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 41187 } } }
I20260806 06:30:17.855120 16114 sys_catalog.cc:458] T 00000000000000000000000000000000 P bbb2a969c88948d2848d8e7758b933c5 [sys.catalog]: This master's current role is: LEADER
I20260806 06:30:17.855283 16115 sys_catalog.cc:455] T 00000000000000000000000000000000 P bbb2a969c88948d2848d8e7758b933c5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bbb2a969c88948d2848d8e7758b933c5. Latest consensus state: current_term: 1 leader_uuid: "bbb2a969c88948d2848d8e7758b933c5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbb2a969c88948d2848d8e7758b933c5" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 41187 } } }
I20260806 06:30:17.855347 16122 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 06:30:17.855352 16115 sys_catalog.cc:458] T 00000000000000000000000000000000 P bbb2a969c88948d2848d8e7758b933c5 [sys.catalog]: This master's current role is: LEADER
I20260806 06:30:17.855901 16122 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 06:30:17.856890 21554 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHhLhT/build/release/bin/kudu as pid 16003
I20260806 06:30:17.857030 21554 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785997581049705-21554-0/minicluster-data/master-0/wal/instance
I20260806 06:30:17.857257 21554 external_mini_cluster.cc:1428] Setting key 5639e61165e90311d2498e9b033f9524
I20260806 06:30:17.857822 16122 catalog_manager.cc:1347] Generated new cluster ID: 2eac183e0c9b4e8fa36a9196a50023e3
I20260806 06:30:17.857878 16122 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 06 06:30:17 dist-test-slave-4tdj krb5kdc[15988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997817, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.12.190@KRBTEST.COM
I20260806 06:30:17.869127 16137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:17.857722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56128 (local address 127.21.12.190:41187)
0806 06:30:17.858410 (+ 688us) server_negotiation.cc:206] Beginning negotiation
0806 06:30:17.858420 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 06:30:17.858454 (+ 34us) server_negotiation.cc:407] Connection header received
0806 06:30:17.858578 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:30:17.858585 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:30:17.858665 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:30:17.858805 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:30:17.859591 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:17.860987 (+ 1396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:17.862007 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:17.862680 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:17.862791 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:30:17.865538 (+ 2747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:30:17.865543 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:30:17.865543 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 06:30:17.865556 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:30:17.867530 (+ 1974us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:30:17.867919 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:30:17.867921 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:30:17.867922 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:30:17.868024 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:30:17.868213 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:30:17.868213 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:30:17.868214 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:30:17.868501 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:30:17.868621 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0806 06:30:17.868755 (+ 134us) server_negotiation.cc:299] Negotiation successful
0806 06:30:17.868885 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":641,"thread_start_us":80,"threads_started":1}
I20260806 06:30:17.883567 16122 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 06:30:17.884087 16122 catalog_manager.cc:1504] Loading token signing keys...
I20260806 06:30:17.887804 16122 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bbb2a969c88948d2848d8e7758b933c5: Generated new TSK 0
I20260806 06:30:17.887979 16122 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.12.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.12.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.12.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.129.keytab.
Entry for principal kudu/127.21.12.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.129.keytab.
I20260806 06:30:17.912662 21554 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHhLhT/build/release/bin/kudu
/tmp/dist-test-taskFHhLhT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785997581049705-21554-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785997581049705-21554-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785997581049705-21554-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785997581049705-21554-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.12.129:0
--local_ip_for_outbound_sockets=127.21.12.129
--webserver_interface=127.21.12.129
--webserver_port=0
--tserver_master_addrs=127.21.12.190:41187
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.129.keytab
--principal=kudu/127.21.12.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.148:41159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785997581049705-21554-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785997581049705-21554-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785997581049705-21554-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 06:30:17.991827 16141 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 06:30:17.992020 16141 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 06:30:17.992041 16141 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 06:30:17.993121 16141 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 06:30:17.993153 16141 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 06:30:17.993177 16141 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.12.129
I20260806 06:30:17.994589 16141 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.148:41159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785997581049705-21554-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785997581049705-21554-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.129.keytab
--principal=kudu/127.21.12.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.12.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785997581049705-21554-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.12.129
--webserver_port=0
--tserver_master_addrs=127.21.12.190:41187
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16141
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.12.129
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785997581049705-21554-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 05:11:02 UTC on ccycloud.quasar-hvzbbz.root.comops.site
build id 4569617
I20260806 06:30:17.994773 16141 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 06:30:17.994940 16141 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 06:30:17.999650 16148 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:30:17.999572 16149 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:30:17.999766 16141 server_base.cc:1034] running on GCE node
W20260806 06:30:17.999575 16151 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:30:18.000042 16141 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 06:30:18.000270 16141 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 06:30:18.001422 16141 hybrid_clock.cc:648] HybridClock initialized: now 1785997818001411 us; error 29 us; skew 500 ppm
Aug 06 06:30:17 dist-test-slave-4tdj krb5kdc[15988](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997817, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 06:30:18.003774 16141 init.cc:377] Logged in from keytab as kudu/127.21.12.129@KRBTEST.COM (short username kudu)
I20260806 06:30:18.004164 16141 webserver.cc:460] Webserver started at http://127.21.12.129:42313/ using document root <none> and password file <none>
I20260806 06:30:18.004359 16141 fs_manager.cc:359] Metadata directory not provided
I20260806 06:30:18.004406 16141 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 06:30:18.004526 16141 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 06:30:18.005527 16141 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785997581049705-21554-0/minicluster-data/ts-0/data/instance:
uuid: "3ccdb3a132f74d1cba733cfec08f5583"
format_stamp: "Formatted at 2026-08-06 06:30:17 on dist-test-slave-4tdj"
server_key: "6be6e3a5b9b1e6e91043a2abf2049ff4"
server_key_iv: "be4bc674206f2c9bacd849905ae5ff1a"
server_key_version: "encryptionkey@0"
I20260806 06:30:18.005852 16141 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785997581049705-21554-0/minicluster-data/ts-0/wal/instance:
uuid: "3ccdb3a132f74d1cba733cfec08f5583"
format_stamp: "Formatted at 2026-08-06 06:30:17 on dist-test-slave-4tdj"
server_key: "6be6e3a5b9b1e6e91043a2abf2049ff4"
server_key_iv: "be4bc674206f2c9bacd849905ae5ff1a"
server_key_version: "encryptionkey@0"
I20260806 06:30:18.007109 16141 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260806 06:30:18.007758 16161 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 06:30:18.007905 16141 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260806 06:30:18.007989 16141 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785997581049705-21554-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785997581049705-21554-0/minicluster-data/ts-0/wal
uuid: "3ccdb3a132f74d1cba733cfec08f5583"
format_stamp: "Formatted at 2026-08-06 06:30:17 on dist-test-slave-4tdj"
server_key: "6be6e3a5b9b1e6e91043a2abf2049ff4"
server_key_iv: "be4bc674206f2c9bacd849905ae5ff1a"
server_key_version: "encryptionkey@0"
I20260806 06:30:18.008044 16141 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785997581049705-21554-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785997581049705-21554-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785997581049705-21554-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 06:30:18.022080 16141 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 06:30:18.023913 16141 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 06:30:18.024042 16141 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 06:30:18.024245 16141 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 06:30:18.024521 16141 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 06:30:18.024555 16141 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 06:30:18.024595 16141 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 06:30:18.024618 16141 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 06:30:18.030133 16141 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.129:33727
I20260806 06:30:18.030190 16319 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.129:33727 every 8 connection(s)
I20260806 06:30:18.030570 16141 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785997581049705-21554-0/minicluster-data/ts-0/data/info.pb
Aug 06 06:30:18 dist-test-slave-4tdj krb5kdc[15988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997817, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.129@KRBTEST.COM for kudu/127.21.12.190@KRBTEST.COM
I20260806 06:30:18.037082 21554 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHhLhT/build/release/bin/kudu as pid 16141
I20260806 06:30:18.037197 21554 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785997581049705-21554-0/minicluster-data/ts-0/wal/instance
I20260806 06:30:18.037385 21554 external_mini_cluster.cc:1428] Setting key 41ccc98f939bccc33a698881d82eb5de
I20260806 06:30:18.039424 16137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:18.031255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.129:35293 (local address 127.21.12.190:41187)
0806 06:30:18.031324 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0806 06:30:18.031326 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:30:18.031517 (+ 191us) server_negotiation.cc:407] Connection header received
0806 06:30:18.032028 (+ 511us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:30:18.032030 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:30:18.032046 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:30:18.032102 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:30:18.033017 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:18.033521 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:18.034492 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:18.034709 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:18.034767 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:30:18.036694 (+ 1927us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:30:18.036695 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:30:18.036696 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 06:30:18.036705 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:30:18.038154 (+ 1449us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:30:18.038579 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:30:18.038580 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:30:18.038580 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:30:18.038631 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:30:18.038871 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:30:18.038872 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:30:18.038872 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:30:18.038963 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:30:18.039005 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0806 06:30:18.039281 (+ 276us) server_negotiation.cc:299] Negotiation successful
0806 06:30:18.039358 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260806 06:30:18.039629 16323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:18.031215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:41187 (local address 127.21.12.129:35293)
0806 06:30:18.031385 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0806 06:30:18.031438 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0806 06:30:18.031937 (+ 499us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:30:18.032172 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:30:18.032176 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:30:18.032744 (+ 568us) client_negotiation.cc:189] Negotiated authn=SASL
0806 06:30:18.032925 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:30:18.032929 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:30:18.033611 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:30:18.033613 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:30:18.034418 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:30:18.034421 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:30:18.034778 (+ 357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:30:18.034779 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:30:18.034892 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:30:18.035272 (+ 380us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 06:30:18.035280 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0806 06:30:18.036590 (+ 1310us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 06:30:18.038225 (+ 1635us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:30:18.038234 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:30:18.038236 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:30:18.038514 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:30:18.038685 (+ 171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:30:18.038686 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:30:18.038686 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:30:18.038817 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:30:18.039008 (+ 191us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 06:30:18.039013 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 06:30:18.039159 (+ 146us) client_negotiation.cc:769] Sending connection context
0806 06:30:18.039259 (+ 100us) client_negotiation.cc:240] Negotiation successful
0806 06:30:18.039410 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":93,"threads_started":1}
I20260806 06:30:18.040283 16320 heartbeater.cc:347] Connected to a master server at 127.21.12.190:41187
I20260806 06:30:18.040365 16320 heartbeater.cc:464] Registering TS with master...
I20260806 06:30:18.040530 16320 heartbeater.cc:511] Master 127.21.12.190:41187 requested a full tablet report, sending...
I20260806 06:30:18.041075 16059 ts_manager.cc:194] Registered new tserver with Master: 3ccdb3a132f74d1cba733cfec08f5583 (127.21.12.129:33727)
I20260806 06:30:18.041672 16059 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.12.129@KRBTEST.COM'} at 127.21.12.129:35293
WARNING: no policy specified for kudu/127.21.12.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.12.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.12.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.130.keytab.
Entry for principal kudu/127.21.12.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.130.keytab.
I20260806 06:30:18.060550 21554 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHhLhT/build/release/bin/kudu
/tmp/dist-test-taskFHhLhT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785997581049705-21554-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785997581049705-21554-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785997581049705-21554-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785997581049705-21554-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.12.130:0
--local_ip_for_outbound_sockets=127.21.12.130
--webserver_interface=127.21.12.130
--webserver_port=0
--tserver_master_addrs=127.21.12.190:41187
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.130.keytab
--principal=kudu/127.21.12.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.148:41159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785997581049705-21554-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785997581049705-21554-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785997581049705-21554-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 06:30:18.139009 16326 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 06:30:18.139211 16326 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 06:30:18.139242 16326 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 06:30:18.140336 16326 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 06:30:18.140374 16326 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 06:30:18.140407 16326 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.12.130
I20260806 06:30:18.141837 16326 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.148:41159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785997581049705-21554-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785997581049705-21554-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.130.keytab
--principal=kudu/127.21.12.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.12.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785997581049705-21554-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.12.130
--webserver_port=0
--tserver_master_addrs=127.21.12.190:41187
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16326
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.12.130
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785997581049705-21554-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 05:11:02 UTC on ccycloud.quasar-hvzbbz.root.comops.site
build id 4569617
I20260806 06:30:18.142057 16326 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 06:30:18.142253 16326 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 06:30:18.147925 16335 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:30:18.147876 16340 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:30:18.148113 16326 server_base.cc:1034] running on GCE node
W20260806 06:30:18.148229 16336 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:30:18.148409 16326 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 06:30:18.148641 16326 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 06:30:18.149777 16326 hybrid_clock.cc:648] HybridClock initialized: now 1785997818149765 us; error 28 us; skew 500 ppm
Aug 06 06:30:18 dist-test-slave-4tdj krb5kdc[15988](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997818, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 06:30:18.152204 16326 init.cc:377] Logged in from keytab as kudu/127.21.12.130@KRBTEST.COM (short username kudu)
I20260806 06:30:18.152590 16326 webserver.cc:460] Webserver started at http://127.21.12.130:33587/ using document root <none> and password file <none>
I20260806 06:30:18.152761 16326 fs_manager.cc:359] Metadata directory not provided
I20260806 06:30:18.152799 16326 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 06:30:18.152889 16326 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 06:30:18.153872 16326 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785997581049705-21554-0/minicluster-data/ts-1/data/instance:
uuid: "3cd8c3ccfba54fd890e5266c7c062880"
format_stamp: "Formatted at 2026-08-06 06:30:18 on dist-test-slave-4tdj"
server_key: "08e121af29167165194d94f0c05997e9"
server_key_iv: "a4d6cace2b91caab723f9a9a4e6956ee"
server_key_version: "encryptionkey@0"
I20260806 06:30:18.154232 16326 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785997581049705-21554-0/minicluster-data/ts-1/wal/instance:
uuid: "3cd8c3ccfba54fd890e5266c7c062880"
format_stamp: "Formatted at 2026-08-06 06:30:18 on dist-test-slave-4tdj"
server_key: "08e121af29167165194d94f0c05997e9"
server_key_iv: "a4d6cace2b91caab723f9a9a4e6956ee"
server_key_version: "encryptionkey@0"
I20260806 06:30:18.155607 16326 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260806 06:30:18.156276 16350 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 06:30:18.156405 16326 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260806 06:30:18.156486 16326 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785997581049705-21554-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785997581049705-21554-0/minicluster-data/ts-1/wal
uuid: "3cd8c3ccfba54fd890e5266c7c062880"
format_stamp: "Formatted at 2026-08-06 06:30:18 on dist-test-slave-4tdj"
server_key: "08e121af29167165194d94f0c05997e9"
server_key_iv: "a4d6cace2b91caab723f9a9a4e6956ee"
server_key_version: "encryptionkey@0"
I20260806 06:30:18.156551 16326 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785997581049705-21554-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785997581049705-21554-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785997581049705-21554-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 06:30:18.172073 16326 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 06:30:18.174114 16326 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 06:30:18.174242 16326 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 06:30:18.174467 16326 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 06:30:18.174767 16326 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 06:30:18.174800 16326 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 06:30:18.174821 16326 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 06:30:18.174842 16326 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 06:30:18.180255 16326 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.130:41109
I20260806 06:30:18.180298 16514 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.130:41109 every 8 connection(s)
I20260806 06:30:18.180578 16326 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785997581049705-21554-0/minicluster-data/ts-1/data/info.pb
I20260806 06:30:18.184849 21554 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHhLhT/build/release/bin/kudu as pid 16326
I20260806 06:30:18.184950 21554 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785997581049705-21554-0/minicluster-data/ts-1/wal/instance
I20260806 06:30:18.185151 21554 external_mini_cluster.cc:1428] Setting key 22cb0b85033c5b4f3367bedaea73bdc3
Aug 06 06:30:18 dist-test-slave-4tdj krb5kdc[15988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997818, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.130@KRBTEST.COM for kudu/127.21.12.190@KRBTEST.COM
I20260806 06:30:18.189568 16137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:18.181232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.130:47491 (local address 127.21.12.190:41187)
0806 06:30:18.181284 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0806 06:30:18.181286 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:30:18.181468 (+ 182us) server_negotiation.cc:407] Connection header received
0806 06:30:18.181975 (+ 507us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:30:18.181976 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:30:18.181990 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:30:18.182041 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:30:18.182996 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:18.183502 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:18.184443 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:18.184638 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:18.184695 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:30:18.187020 (+ 2325us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:30:18.187021 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:30:18.187022 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 06:30:18.187028 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:30:18.188354 (+ 1326us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:30:18.188729 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:30:18.188731 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:30:18.188731 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:30:18.188794 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:30:18.188988 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:30:18.188989 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:30:18.188989 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:30:18.189124 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:30:18.189190 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0806 06:30:18.189397 (+ 207us) server_negotiation.cc:299] Negotiation successful
0806 06:30:18.189484 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260806 06:30:18.189651 16517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:18.181191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:41187 (local address 127.21.12.130:47491)
0806 06:30:18.181354 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0806 06:30:18.181390 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0806 06:30:18.181880 (+ 490us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:30:18.182119 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:30:18.182123 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:30:18.182716 (+ 593us) client_negotiation.cc:189] Negotiated authn=SASL
0806 06:30:18.182906 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:30:18.182910 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:30:18.183582 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:30:18.183583 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:30:18.184362 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:30:18.184364 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:30:18.184722 (+ 358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:30:18.184723 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:30:18.184830 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:30:18.185313 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 06:30:18.185323 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0806 06:30:18.186932 (+ 1609us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 06:30:18.188439 (+ 1507us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:30:18.188446 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:30:18.188448 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:30:18.188666 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:30:18.188852 (+ 186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:30:18.188852 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:30:18.188853 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:30:18.188938 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:30:18.189180 (+ 242us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 06:30:18.189183 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 06:30:18.189289 (+ 106us) client_negotiation.cc:769] Sending connection context
0806 06:30:18.189362 (+ 73us) client_negotiation.cc:240] Negotiation successful
0806 06:30:18.189475 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":84,"threads_started":1}
I20260806 06:30:18.190078 16515 heartbeater.cc:347] Connected to a master server at 127.21.12.190:41187
I20260806 06:30:18.190165 16515 heartbeater.cc:464] Registering TS with master...
I20260806 06:30:18.190380 16515 heartbeater.cc:511] Master 127.21.12.190:41187 requested a full tablet report, sending...
I20260806 06:30:18.190757 16059 ts_manager.cc:194] Registered new tserver with Master: 3cd8c3ccfba54fd890e5266c7c062880 (127.21.12.130:41109)
I20260806 06:30:18.191246 16059 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.12.130@KRBTEST.COM'} at 127.21.12.130:47491
WARNING: no policy specified for kudu/127.21.12.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.12.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.12.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.131.keytab.
Entry for principal kudu/127.21.12.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.131.keytab.
I20260806 06:30:18.208657 21554 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHhLhT/build/release/bin/kudu
/tmp/dist-test-taskFHhLhT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785997581049705-21554-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785997581049705-21554-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785997581049705-21554-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785997581049705-21554-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.12.131:0
--local_ip_for_outbound_sockets=127.21.12.131
--webserver_interface=127.21.12.131
--webserver_port=0
--tserver_master_addrs=127.21.12.190:41187
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.131.keytab
--principal=kudu/127.21.12.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.148:41159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785997581049705-21554-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785997581049705-21554-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785997581049705-21554-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 06:30:18.287865 16520 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 06:30:18.288057 16520 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 06:30:18.288089 16520 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 06:30:18.289201 16520 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 06:30:18.289242 16520 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 06:30:18.289274 16520 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.12.131
I20260806 06:30:18.290736 16520 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.148:41159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785997581049705-21554-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785997581049705-21554-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.131.keytab
--principal=kudu/127.21.12.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.12.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785997581049705-21554-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.12.131
--webserver_port=0
--tserver_master_addrs=127.21.12.190:41187
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16520
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.12.131
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785997581049705-21554-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 05:11:02 UTC on ccycloud.quasar-hvzbbz.root.comops.site
build id 4569617
I20260806 06:30:18.290967 16520 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 06:30:18.291172 16520 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 06:30:18.295903 16531 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:30:18.295915 16527 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:30:18.296092 16520 server_base.cc:1034] running on GCE node
W20260806 06:30:18.295918 16528 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:30:18.296326 16520 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 06:30:18.296548 16520 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 06:30:18.297693 16520 hybrid_clock.cc:648] HybridClock initialized: now 1785997818297681 us; error 28 us; skew 500 ppm
Aug 06 06:30:18 dist-test-slave-4tdj krb5kdc[15988](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997818, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 06:30:18.300073 16520 init.cc:377] Logged in from keytab as kudu/127.21.12.131@KRBTEST.COM (short username kudu)
I20260806 06:30:18.300479 16520 webserver.cc:460] Webserver started at http://127.21.12.131:43011/ using document root <none> and password file <none>
I20260806 06:30:18.300699 16520 fs_manager.cc:359] Metadata directory not provided
I20260806 06:30:18.300748 16520 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 06:30:18.300870 16520 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 06:30:18.301858 16520 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785997581049705-21554-0/minicluster-data/ts-2/data/instance:
uuid: "57dacd8d2f9844388116c58f60d8226b"
format_stamp: "Formatted at 2026-08-06 06:30:18 on dist-test-slave-4tdj"
server_key: "c863080c92a1fe87e1a24be11c0436bd"
server_key_iv: "084482dfc7b71876d8a68733a493c7b0"
server_key_version: "encryptionkey@0"
I20260806 06:30:18.302174 16520 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785997581049705-21554-0/minicluster-data/ts-2/wal/instance:
uuid: "57dacd8d2f9844388116c58f60d8226b"
format_stamp: "Formatted at 2026-08-06 06:30:18 on dist-test-slave-4tdj"
server_key: "c863080c92a1fe87e1a24be11c0436bd"
server_key_iv: "084482dfc7b71876d8a68733a493c7b0"
server_key_version: "encryptionkey@0"
I20260806 06:30:18.303406 16520 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260806 06:30:18.304045 16541 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 06:30:18.304188 16520 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260806 06:30:18.304277 16520 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785997581049705-21554-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785997581049705-21554-0/minicluster-data/ts-2/wal
uuid: "57dacd8d2f9844388116c58f60d8226b"
format_stamp: "Formatted at 2026-08-06 06:30:18 on dist-test-slave-4tdj"
server_key: "c863080c92a1fe87e1a24be11c0436bd"
server_key_iv: "084482dfc7b71876d8a68733a493c7b0"
server_key_version: "encryptionkey@0"
I20260806 06:30:18.304351 16520 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785997581049705-21554-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785997581049705-21554-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785997581049705-21554-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 06:30:18.334771 16520 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 06:30:18.336671 16520 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 06:30:18.336817 16520 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 06:30:18.337038 16520 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 06:30:18.337344 16520 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 06:30:18.337379 16520 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 06:30:18.337409 16520 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 06:30:18.337426 16520 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 06:30:18.342955 16520 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.131:44469
I20260806 06:30:18.343016 16694 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.131:44469 every 8 connection(s)
I20260806 06:30:18.343343 16520 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785997581049705-21554-0/minicluster-data/ts-2/data/info.pb
Aug 06 06:30:18 dist-test-slave-4tdj krb5kdc[15988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997818, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.131@KRBTEST.COM for kudu/127.21.12.190@KRBTEST.COM
I20260806 06:30:18.351696 16137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:18.344064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.131:33233 (local address 127.21.12.190:41187)
0806 06:30:18.344112 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0806 06:30:18.344114 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:30:18.344312 (+ 198us) server_negotiation.cc:407] Connection header received
0806 06:30:18.344798 (+ 486us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:30:18.344799 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:30:18.344808 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:30:18.344858 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:30:18.345732 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:18.346219 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:18.347177 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:18.347352 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:18.347397 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:30:18.349253 (+ 1856us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:30:18.349254 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:30:18.349255 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 06:30:18.349263 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:30:18.350495 (+ 1232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:30:18.350854 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:30:18.350855 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:30:18.350856 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:30:18.350907 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:30:18.351118 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:30:18.351119 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:30:18.351119 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:30:18.351223 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:30:18.351279 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0806 06:30:18.351530 (+ 251us) server_negotiation.cc:299] Negotiation successful
0806 06:30:18.351606 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260806 06:30:18.351866 16697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:18.344040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:41187 (local address 127.21.12.131:33233)
0806 06:30:18.344204 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0806 06:30:18.344245 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0806 06:30:18.344723 (+ 478us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:30:18.344912 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:30:18.344916 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:30:18.345466 (+ 550us) client_negotiation.cc:189] Negotiated authn=SASL
0806 06:30:18.345667 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:30:18.345672 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:30:18.346284 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:30:18.346284 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:30:18.347114 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:30:18.347116 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:30:18.347404 (+ 288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:30:18.347405 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:30:18.347511 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:30:18.347885 (+ 374us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 06:30:18.347892 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0806 06:30:18.349162 (+ 1270us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 06:30:18.350571 (+ 1409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:30:18.350578 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:30:18.350580 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:30:18.350790 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:30:18.350959 (+ 169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:30:18.350960 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:30:18.350960 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:30:18.351061 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:30:18.351277 (+ 216us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 06:30:18.351284 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 06:30:18.351412 (+ 128us) client_negotiation.cc:769] Sending connection context
0806 06:30:18.351498 (+ 86us) client_negotiation.cc:240] Negotiation successful
0806 06:30:18.351658 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":74,"threads_started":1}
I20260806 06:30:18.352371 16695 heartbeater.cc:347] Connected to a master server at 127.21.12.190:41187
I20260806 06:30:18.352461 16695 heartbeater.cc:464] Registering TS with master...
I20260806 06:30:18.352627 16695 heartbeater.cc:511] Master 127.21.12.190:41187 requested a full tablet report, sending...
I20260806 06:30:18.352962 16059 ts_manager.cc:194] Registered new tserver with Master: 57dacd8d2f9844388116c58f60d8226b (127.21.12.131:44469)
I20260806 06:30:18.353322 16059 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.12.131@KRBTEST.COM'} at 127.21.12.131:33233
I20260806 06:30:18.353539 21554 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHhLhT/build/release/bin/kudu as pid 16520
I20260806 06:30:18.353654 21554 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785997581049705-21554-0/minicluster-data/ts-2/wal/instance
I20260806 06:30:18.353890 21554 external_mini_cluster.cc:1428] Setting key e2492226b88bd4adcb8861cb362e1c97
I20260806 06:30:18.354888 21554 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 06:30:18.361914 16137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:18.356260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56136 (local address 127.21.12.190:41187)
0806 06:30:18.356305 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0806 06:30:18.356307 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:30:18.356413 (+ 106us) server_negotiation.cc:407] Connection header received
0806 06:30:18.356479 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:30:18.356480 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:30:18.356491 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:30:18.356548 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:30:18.357100 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:18.357580 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:18.358327 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:18.358532 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:18.358586 (+ 54us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:30:18.359784 (+ 1198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:30:18.359785 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:30:18.359786 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 06:30:18.359792 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:30:18.360947 (+ 1155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:30:18.361320 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:30:18.361321 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:30:18.361321 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:30:18.361379 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:30:18.361517 (+ 138us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:30:18.361518 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:30:18.361518 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:30:18.361634 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:30:18.361718 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0806 06:30:18.361805 (+ 87us) server_negotiation.cc:299] Negotiation successful
0806 06:30:18.361856 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260806 06:30:18.363349 16059 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56136:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 06:30:18.364832 16059 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 06:30:18.371137 16711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:18.367187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42096 (local address 127.21.12.129:33727)
0806 06:30:18.367707 (+ 520us) server_negotiation.cc:206] Beginning negotiation
0806 06:30:18.367710 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 06:30:18.367721 (+ 11us) server_negotiation.cc:407] Connection header received
0806 06:30:18.367760 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:30:18.367762 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:30:18.367778 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:30:18.367869 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 06:30:18.367985 (+ 116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:18.368952 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:18.369908 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:18.370679 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:18.370760 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:30:18.370825 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0806 06:30:18.371010 (+ 185us) server_negotiation.cc:299] Negotiation successful
0806 06:30:18.371056 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":477,"thread_start_us":83,"threads_started":1}
I20260806 06:30:18.371146 16708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:18.367154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56524 (local address 127.21.12.131:44469)
0806 06:30:18.367334 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0806 06:30:18.367338 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 06:30:18.367346 (+ 8us) server_negotiation.cc:407] Connection header received
0806 06:30:18.367374 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:30:18.367376 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:30:18.367387 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:30:18.367453 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 06:30:18.367621 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:18.368674 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:18.370338 (+ 1664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:18.370829 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:18.370895 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:30:18.370947 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0806 06:30:18.371034 (+ 87us) server_negotiation.cc:299] Negotiation successful
0806 06:30:18.371069 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":102,"threads_started":1}
I20260806 06:30:18.371135 16704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:18.366842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.130:41109 (local address 127.0.0.1:40392)
0806 06:30:18.366979 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0806 06:30:18.367017 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0806 06:30:18.367080 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:30:18.367568 (+ 488us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:30:18.367569 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:30:18.367570 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 06:30:18.367611 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:30:18.367612 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:30:18.368653 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:30:18.368654 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:30:18.369933 (+ 1279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:30:18.369935 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:30:18.370816 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:30:18.370817 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:30:18.370883 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:30:18.370906 (+ 23us) client_negotiation.cc:769] Sending connection context
0806 06:30:18.370945 (+ 39us) client_negotiation.cc:240] Negotiation successful
0806 06:30:18.371038 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"mutex_wait_us":28,"thread_start_us":68,"threads_started":1}
I20260806 06:30:18.371284 16705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:18.366867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.131:44469 (local address 127.0.0.1:56524)
0806 06:30:18.367204 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0806 06:30:18.367213 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0806 06:30:18.367248 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:30:18.367473 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:30:18.367476 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:30:18.367478 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 06:30:18.367544 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:30:18.367546 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:30:18.368769 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:30:18.368770 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:30:18.369932 (+ 1162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:30:18.369935 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:30:18.370900 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:30:18.370901 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:30:18.370955 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:30:18.370958 (+ 3us) client_negotiation.cc:769] Sending connection context
0806 06:30:18.371023 (+ 65us) client_negotiation.cc:240] Negotiation successful
0806 06:30:18.371053 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":326,"mutex_wait_us":5,"thread_start_us":66,"threads_started":1}
I20260806 06:30:18.371205 16703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:18.367047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.129:33727 (local address 127.0.0.1:42096)
0806 06:30:18.367339 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0806 06:30:18.367351 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 06:30:18.367406 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:30:18.367873 (+ 467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:30:18.367874 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:30:18.367876 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 06:30:18.367907 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:30:18.367909 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:30:18.369083 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:30:18.369085 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:30:18.369816 (+ 731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:30:18.369818 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:30:18.370788 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:30:18.370789 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:30:18.370877 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:30:18.370895 (+ 18us) client_negotiation.cc:769] Sending connection context
0806 06:30:18.370956 (+ 61us) client_negotiation.cc:240] Negotiation successful
0806 06:30:18.371095 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"mutex_wait_us":99,"thread_start_us":42,"threads_started":1}
I20260806 06:30:18.371845 16706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:18.367021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40392 (local address 127.21.12.130:41109)
0806 06:30:18.367407 (+ 386us) server_negotiation.cc:206] Beginning negotiation
0806 06:30:18.367412 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 06:30:18.367423 (+ 11us) server_negotiation.cc:407] Connection header received
0806 06:30:18.367464 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:30:18.367466 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:30:18.367484 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:30:18.367558 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 06:30:18.367666 (+ 108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:18.368567 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:18.370015 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:18.370713 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:18.371291 (+ 578us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:30:18.371681 (+ 390us) server_negotiation.cc:1035] Waiting for connection context
0806 06:30:18.371712 (+ 31us) server_negotiation.cc:299] Negotiation successful
0806 06:30:18.371757 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":104,"threads_started":1}
I20260806 06:30:18.372061 16606 tablet_service.cc:1467] Processing CreateTablet for tablet 1db967605c1c4e30ad1d30533effb17a (DEFAULT_TABLE table=test-table [id=89c8f36547524628994a99564e3b0938]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 06:30:18.372066 16225 tablet_service.cc:1467] Processing CreateTablet for tablet 1db967605c1c4e30ad1d30533effb17a (DEFAULT_TABLE table=test-table [id=89c8f36547524628994a99564e3b0938]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 06:30:18.372434 16225 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1db967605c1c4e30ad1d30533effb17a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 06:30:18.372437 16606 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1db967605c1c4e30ad1d30533effb17a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 06:30:18.372517 16418 tablet_service.cc:1467] Processing CreateTablet for tablet 1db967605c1c4e30ad1d30533effb17a (DEFAULT_TABLE table=test-table [id=89c8f36547524628994a99564e3b0938]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 06:30:18.372824 16418 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1db967605c1c4e30ad1d30533effb17a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 06:30:18.375010 16713 tablet_bootstrap.cc:492] T 1db967605c1c4e30ad1d30533effb17a P 3ccdb3a132f74d1cba733cfec08f5583: Bootstrap starting.
I20260806 06:30:18.375548 16713 tablet_bootstrap.cc:654] T 1db967605c1c4e30ad1d30533effb17a P 3ccdb3a132f74d1cba733cfec08f5583: Neither blocks nor log segments found. Creating new log.
I20260806 06:30:18.375892 16715 tablet_bootstrap.cc:492] T 1db967605c1c4e30ad1d30533effb17a P 57dacd8d2f9844388116c58f60d8226b: Bootstrap starting.
I20260806 06:30:18.376194 16713 log.cc:824] T 1db967605c1c4e30ad1d30533effb17a P 3ccdb3a132f74d1cba733cfec08f5583: Log is configured to *not* fsync() on all Append() calls
I20260806 06:30:18.376204 16716 tablet_bootstrap.cc:492] T 1db967605c1c4e30ad1d30533effb17a P 3cd8c3ccfba54fd890e5266c7c062880: Bootstrap starting.
I20260806 06:30:18.376693 16715 tablet_bootstrap.cc:654] T 1db967605c1c4e30ad1d30533effb17a P 57dacd8d2f9844388116c58f60d8226b: Neither blocks nor log segments found. Creating new log.
I20260806 06:30:18.376807 16716 tablet_bootstrap.cc:654] T 1db967605c1c4e30ad1d30533effb17a P 3cd8c3ccfba54fd890e5266c7c062880: Neither blocks nor log segments found. Creating new log.
I20260806 06:30:18.376860 16713 tablet_bootstrap.cc:492] T 1db967605c1c4e30ad1d30533effb17a P 3ccdb3a132f74d1cba733cfec08f5583: No bootstrap required, opened a new log
I20260806 06:30:18.376927 16713 ts_tablet_manager.cc:1397] T 1db967605c1c4e30ad1d30533effb17a P 3ccdb3a132f74d1cba733cfec08f5583: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260806 06:30:18.377281 16715 log.cc:824] T 1db967605c1c4e30ad1d30533effb17a P 57dacd8d2f9844388116c58f60d8226b: Log is configured to *not* fsync() on all Append() calls
I20260806 06:30:18.377336 16716 log.cc:824] T 1db967605c1c4e30ad1d30533effb17a P 3cd8c3ccfba54fd890e5266c7c062880: Log is configured to *not* fsync() on all Append() calls
I20260806 06:30:18.378031 16715 tablet_bootstrap.cc:492] T 1db967605c1c4e30ad1d30533effb17a P 57dacd8d2f9844388116c58f60d8226b: No bootstrap required, opened a new log
I20260806 06:30:18.378077 16716 tablet_bootstrap.cc:492] T 1db967605c1c4e30ad1d30533effb17a P 3cd8c3ccfba54fd890e5266c7c062880: No bootstrap required, opened a new log
I20260806 06:30:18.378098 16715 ts_tablet_manager.cc:1397] T 1db967605c1c4e30ad1d30533effb17a P 57dacd8d2f9844388116c58f60d8226b: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260806 06:30:18.378135 16716 ts_tablet_manager.cc:1397] T 1db967605c1c4e30ad1d30533effb17a P 3cd8c3ccfba54fd890e5266c7c062880: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260806 06:30:18.379037 16713 raft_consensus.cc:348] T 1db967605c1c4e30ad1d30533effb17a P 3ccdb3a132f74d1cba733cfec08f5583 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cd8c3ccfba54fd890e5266c7c062880" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 41109 } } peers { permanent_uuid: "57dacd8d2f9844388116c58f60d8226b" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 44469 } } peers { permanent_uuid: "3ccdb3a132f74d1cba733cfec08f5583" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 33727 } }
I20260806 06:30:18.379225 16713 raft_consensus.cc:374] T 1db967605c1c4e30ad1d30533effb17a P 3ccdb3a132f74d1cba733cfec08f5583 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 06:30:18.379266 16713 raft_consensus.cc:729] T 1db967605c1c4e30ad1d30533effb17a P 3ccdb3a132f74d1cba733cfec08f5583 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ccdb3a132f74d1cba733cfec08f5583, State: Initialized, Role: FOLLOWER
I20260806 06:30:18.379418 16713 consensus_queue.cc:260] T 1db967605c1c4e30ad1d30533effb17a P 3ccdb3a132f74d1cba733cfec08f5583 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cd8c3ccfba54fd890e5266c7c062880" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 41109 } } peers { permanent_uuid: "57dacd8d2f9844388116c58f60d8226b" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 44469 } } peers { permanent_uuid: "3ccdb3a132f74d1cba733cfec08f5583" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 33727 } }
I20260806 06:30:18.379693 16713 ts_tablet_manager.cc:1428] T 1db967605c1c4e30ad1d30533effb17a P 3ccdb3a132f74d1cba733cfec08f5583: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260806 06:30:18.379626 16715 raft_consensus.cc:348] T 1db967605c1c4e30ad1d30533effb17a P 57dacd8d2f9844388116c58f60d8226b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cd8c3ccfba54fd890e5266c7c062880" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 41109 } } peers { permanent_uuid: "57dacd8d2f9844388116c58f60d8226b" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 44469 } } peers { permanent_uuid: "3ccdb3a132f74d1cba733cfec08f5583" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 33727 } }
I20260806 06:30:18.379727 16320 heartbeater.cc:503] Master 127.21.12.190:41187 was elected leader, sending a full tablet report...
I20260806 06:30:18.379804 16715 raft_consensus.cc:374] T 1db967605c1c4e30ad1d30533effb17a P 57dacd8d2f9844388116c58f60d8226b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 06:30:18.379850 16715 raft_consensus.cc:729] T 1db967605c1c4e30ad1d30533effb17a P 57dacd8d2f9844388116c58f60d8226b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57dacd8d2f9844388116c58f60d8226b, State: Initialized, Role: FOLLOWER
I20260806 06:30:18.379956 16715 consensus_queue.cc:260] T 1db967605c1c4e30ad1d30533effb17a P 57dacd8d2f9844388116c58f60d8226b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cd8c3ccfba54fd890e5266c7c062880" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 41109 } } peers { permanent_uuid: "57dacd8d2f9844388116c58f60d8226b" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 44469 } } peers { permanent_uuid: "3ccdb3a132f74d1cba733cfec08f5583" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 33727 } }
I20260806 06:30:18.379972 16716 raft_consensus.cc:348] T 1db967605c1c4e30ad1d30533effb17a P 3cd8c3ccfba54fd890e5266c7c062880 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cd8c3ccfba54fd890e5266c7c062880" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 41109 } } peers { permanent_uuid: "57dacd8d2f9844388116c58f60d8226b" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 44469 } } peers { permanent_uuid: "3ccdb3a132f74d1cba733cfec08f5583" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 33727 } }
I20260806 06:30:18.380141 16716 raft_consensus.cc:374] T 1db967605c1c4e30ad1d30533effb17a P 3cd8c3ccfba54fd890e5266c7c062880 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 06:30:18.380182 16716 raft_consensus.cc:729] T 1db967605c1c4e30ad1d30533effb17a P 3cd8c3ccfba54fd890e5266c7c062880 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3cd8c3ccfba54fd890e5266c7c062880, State: Initialized, Role: FOLLOWER
I20260806 06:30:18.380254 16715 ts_tablet_manager.cc:1428] T 1db967605c1c4e30ad1d30533effb17a P 57dacd8d2f9844388116c58f60d8226b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260806 06:30:18.380250 16695 heartbeater.cc:503] Master 127.21.12.190:41187 was elected leader, sending a full tablet report...
I20260806 06:30:18.380309 16716 consensus_queue.cc:260] T 1db967605c1c4e30ad1d30533effb17a P 3cd8c3ccfba54fd890e5266c7c062880 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cd8c3ccfba54fd890e5266c7c062880" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 41109 } } peers { permanent_uuid: "57dacd8d2f9844388116c58f60d8226b" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 44469 } } peers { permanent_uuid: "3ccdb3a132f74d1cba733cfec08f5583" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 33727 } }
I20260806 06:30:18.380545 16716 ts_tablet_manager.cc:1428] T 1db967605c1c4e30ad1d30533effb17a P 3cd8c3ccfba54fd890e5266c7c062880: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260806 06:30:18.380784 16515 heartbeater.cc:503] Master 127.21.12.190:41187 was elected leader, sending a full tablet report...
I20260806 06:30:18.409289 16722 raft_consensus.cc:482] T 1db967605c1c4e30ad1d30533effb17a P 57dacd8d2f9844388116c58f60d8226b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 06:30:18.409399 16722 raft_consensus.cc:504] T 1db967605c1c4e30ad1d30533effb17a P 57dacd8d2f9844388116c58f60d8226b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cd8c3ccfba54fd890e5266c7c062880" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 41109 } } peers { permanent_uuid: "57dacd8d2f9844388116c58f60d8226b" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 44469 } } peers { permanent_uuid: "3ccdb3a132f74d1cba733cfec08f5583" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 33727 } }
I20260806 06:30:18.409683 16722 leader_election.cc:288] T 1db967605c1c4e30ad1d30533effb17a P 57dacd8d2f9844388116c58f60d8226b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3cd8c3ccfba54fd890e5266c7c062880 (127.21.12.130:41109), 3ccdb3a132f74d1cba733cfec08f5583 (127.21.12.129:33727)
I20260806 06:30:18.413228 16697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:18.409836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.129:33727 (local address 127.21.12.131:49913)
0806 06:30:18.409885 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0806 06:30:18.409898 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 06:30:18.409948 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:30:18.410091 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:30:18.410092 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:30:18.410093 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 06:30:18.410211 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:30:18.410213 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:30:18.411156 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:30:18.411157 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:30:18.412292 (+ 1135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:30:18.412295 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:30:18.413010 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:30:18.413011 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:30:18.413068 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:30:18.413071 (+ 3us) client_negotiation.cc:769] Sending connection context
0806 06:30:18.413116 (+ 45us) client_negotiation.cc:240] Negotiation successful
0806 06:30:18.413156 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260806 06:30:18.413232 16711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:18.409933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.131:49913 (local address 127.21.12.129:33727)
0806 06:30:18.409974 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0806 06:30:18.409990 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 06:30:18.409998 (+ 8us) server_negotiation.cc:407] Connection header received
0806 06:30:18.410023 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:30:18.410024 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:30:18.410032 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:30:18.410083 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 06:30:18.410266 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:18.411087 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:18.412376 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:18.412940 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:18.413014 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:30:18.413024 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0806 06:30:18.413140 (+ 116us) server_negotiation.cc:299] Negotiation successful
0806 06:30:18.413178 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260806 06:30:18.413340 16726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:18.409837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.130:41109 (local address 127.21.12.131:49643)
0806 06:30:18.410335 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0806 06:30:18.410345 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0806 06:30:18.410389 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:30:18.410491 (+ 102us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:30:18.410492 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:30:18.410493 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 06:30:18.410539 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:30:18.410540 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:30:18.411372 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:30:18.411373 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:30:18.412448 (+ 1075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:30:18.412449 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:30:18.413143 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:30:18.413144 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:30:18.413198 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:30:18.413201 (+ 3us) client_negotiation.cc:769] Sending connection context
0806 06:30:18.413247 (+ 46us) client_negotiation.cc:240] Negotiation successful
0806 06:30:18.413278 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":482,"mutex_wait_us":274,"thread_start_us":148,"threads_started":1}
I20260806 06:30:18.413357 16706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:18.409929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.131:49643 (local address 127.21.12.130:41109)
0806 06:30:18.409986 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0806 06:30:18.409988 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:30:18.410389 (+ 401us) server_negotiation.cc:407] Connection header received
0806 06:30:18.410422 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:30:18.410423 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:30:18.410433 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:30:18.410490 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 06:30:18.410578 (+ 88us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:18.411302 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:18.412517 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:18.413066 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:18.413124 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:30:18.413132 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0806 06:30:18.413262 (+ 130us) server_negotiation.cc:299] Negotiation successful
0806 06:30:18.413296 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260806 06:30:18.413684 16453 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1db967605c1c4e30ad1d30533effb17a" candidate_uuid: "57dacd8d2f9844388116c58f60d8226b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3cd8c3ccfba54fd890e5266c7c062880" is_pre_election: true
I20260806 06:30:18.413722 16257 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1db967605c1c4e30ad1d30533effb17a" candidate_uuid: "57dacd8d2f9844388116c58f60d8226b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ccdb3a132f74d1cba733cfec08f5583" is_pre_election: true
I20260806 06:30:18.413823 16453 raft_consensus.cc:2454] T 1db967605c1c4e30ad1d30533effb17a P 3cd8c3ccfba54fd890e5266c7c062880 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57dacd8d2f9844388116c58f60d8226b in term 0.
I20260806 06:30:18.413853 16257 raft_consensus.cc:2454] T 1db967605c1c4e30ad1d30533effb17a P 3ccdb3a132f74d1cba733cfec08f5583 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57dacd8d2f9844388116c58f60d8226b in term 0.
I20260806 06:30:18.414058 16547 leader_election.cc:302] T 1db967605c1c4e30ad1d30533effb17a P 57dacd8d2f9844388116c58f60d8226b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3cd8c3ccfba54fd890e5266c7c062880, 57dacd8d2f9844388116c58f60d8226b; no voters:
I20260806 06:30:18.414243 16722 raft_consensus.cc:2781] T 1db967605c1c4e30ad1d30533effb17a P 57dacd8d2f9844388116c58f60d8226b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 06:30:18.414328 16722 raft_consensus.cc:482] T 1db967605c1c4e30ad1d30533effb17a P 57dacd8d2f9844388116c58f60d8226b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 06:30:18.414362 16722 raft_consensus.cc:3037] T 1db967605c1c4e30ad1d30533effb17a P 57dacd8d2f9844388116c58f60d8226b [term 0 FOLLOWER]: Advancing to term 1
I20260806 06:30:18.415067 16722 raft_consensus.cc:504] T 1db967605c1c4e30ad1d30533effb17a P 57dacd8d2f9844388116c58f60d8226b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cd8c3ccfba54fd890e5266c7c062880" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 41109 } } peers { permanent_uuid: "57dacd8d2f9844388116c58f60d8226b" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 44469 } } peers { permanent_uuid: "3ccdb3a132f74d1cba733cfec08f5583" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 33727 } }
I20260806 06:30:18.415198 16722 leader_election.cc:288] T 1db967605c1c4e30ad1d30533effb17a P 57dacd8d2f9844388116c58f60d8226b [CANDIDATE]: Term 1 election: Requested vote from peers 3cd8c3ccfba54fd890e5266c7c062880 (127.21.12.130:41109), 3ccdb3a132f74d1cba733cfec08f5583 (127.21.12.129:33727)
I20260806 06:30:18.415373 16453 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1db967605c1c4e30ad1d30533effb17a" candidate_uuid: "57dacd8d2f9844388116c58f60d8226b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3cd8c3ccfba54fd890e5266c7c062880"
I20260806 06:30:18.415423 16257 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1db967605c1c4e30ad1d30533effb17a" candidate_uuid: "57dacd8d2f9844388116c58f60d8226b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ccdb3a132f74d1cba733cfec08f5583"
I20260806 06:30:18.415462 16453 raft_consensus.cc:3037] T 1db967605c1c4e30ad1d30533effb17a P 3cd8c3ccfba54fd890e5266c7c062880 [term 0 FOLLOWER]: Advancing to term 1
I20260806 06:30:18.415484 16257 raft_consensus.cc:3037] T 1db967605c1c4e30ad1d30533effb17a P 3ccdb3a132f74d1cba733cfec08f5583 [term 0 FOLLOWER]: Advancing to term 1
I20260806 06:30:18.416318 16453 raft_consensus.cc:2454] T 1db967605c1c4e30ad1d30533effb17a P 3cd8c3ccfba54fd890e5266c7c062880 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57dacd8d2f9844388116c58f60d8226b in term 1.
I20260806 06:30:18.416318 16257 raft_consensus.cc:2454] T 1db967605c1c4e30ad1d30533effb17a P 3ccdb3a132f74d1cba733cfec08f5583 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57dacd8d2f9844388116c58f60d8226b in term 1.
I20260806 06:30:18.416517 16547 leader_election.cc:302] T 1db967605c1c4e30ad1d30533effb17a P 57dacd8d2f9844388116c58f60d8226b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3cd8c3ccfba54fd890e5266c7c062880, 57dacd8d2f9844388116c58f60d8226b; no voters:
I20260806 06:30:18.416610 16722 raft_consensus.cc:2781] T 1db967605c1c4e30ad1d30533effb17a P 57dacd8d2f9844388116c58f60d8226b [term 1 FOLLOWER]: Leader election won for term 1
I20260806 06:30:18.416742 16722 raft_consensus.cc:686] T 1db967605c1c4e30ad1d30533effb17a P 57dacd8d2f9844388116c58f60d8226b [term 1 LEADER]: Becoming Leader. State: Replica: 57dacd8d2f9844388116c58f60d8226b, State: Running, Role: LEADER
I20260806 06:30:18.416815 16722 consensus_queue.cc:237] T 1db967605c1c4e30ad1d30533effb17a P 57dacd8d2f9844388116c58f60d8226b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cd8c3ccfba54fd890e5266c7c062880" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 41109 } } peers { permanent_uuid: "57dacd8d2f9844388116c58f60d8226b" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 44469 } } peers { permanent_uuid: "3ccdb3a132f74d1cba733cfec08f5583" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 33727 } }
I20260806 06:30:18.417621 16059 catalog_manager.cc:5495] T 1db967605c1c4e30ad1d30533effb17a P 57dacd8d2f9844388116c58f60d8226b reported cstate change: term changed from 0 to 1, leader changed from <none> to 57dacd8d2f9844388116c58f60d8226b (127.21.12.131). New cstate: current_term: 1 leader_uuid: "57dacd8d2f9844388116c58f60d8226b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cd8c3ccfba54fd890e5266c7c062880" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 41109 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "57dacd8d2f9844388116c58f60d8226b" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 44469 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3ccdb3a132f74d1cba733cfec08f5583" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 33727 } health_report { overall_health: UNKNOWN } } }
I20260806 06:30:18.430318 16708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:18.428179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56538 (local address 127.21.12.131:44469)
0806 06:30:18.428227 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0806 06:30:18.428229 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:30:18.428238 (+ 9us) server_negotiation.cc:407] Connection header received
0806 06:30:18.428306 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:30:18.428307 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:30:18.428323 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:30:18.428378 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 06:30:18.428482 (+ 104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:18.428916 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:18.429628 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:18.429795 (+ 167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:18.429835 (+ 40us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:30:18.429952 (+ 117us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 06:30:18.430058 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 06:30:18.430136 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0806 06:30:18.430217 (+ 81us) server_negotiation.cc:299] Negotiation successful
0806 06:30:18.430226 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
W20260806 06:30:18.431051 16516 tablet.cc:2369] T 1db967605c1c4e30ad1d30533effb17a P 3cd8c3ccfba54fd890e5266c7c062880: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 06:30:18.432005 16257 raft_consensus.cc:1261] T 1db967605c1c4e30ad1d30533effb17a P 3ccdb3a132f74d1cba733cfec08f5583 [term 1 FOLLOWER]: Refusing update from remote peer 57dacd8d2f9844388116c58f60d8226b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 06:30:18.432008 16453 raft_consensus.cc:1261] T 1db967605c1c4e30ad1d30533effb17a P 3cd8c3ccfba54fd890e5266c7c062880 [term 1 FOLLOWER]: Refusing update from remote peer 57dacd8d2f9844388116c58f60d8226b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 06:30:18.432313 16727 consensus_queue.cc:1035] T 1db967605c1c4e30ad1d30533effb17a P 57dacd8d2f9844388116c58f60d8226b [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ccdb3a132f74d1cba733cfec08f5583" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 33727 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 06:30:18.432431 16722 consensus_queue.cc:1035] T 1db967605c1c4e30ad1d30533effb17a P 57dacd8d2f9844388116c58f60d8226b [LEADER]: Connected to new peer: Peer: permanent_uuid: "3cd8c3ccfba54fd890e5266c7c062880" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 41109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 06:30:18.434571 16731 mvcc.cc:205] Tried to move back new op lower bound from 7315447064295251968 to 7315447064235929600. Current Snapshot: MvccSnapshot[applied={T|T < 7315447064295251968}]
I20260806 06:30:18.434989 16734 mvcc.cc:205] Tried to move back new op lower bound from 7315447064295251968 to 7315447064235929600. Current Snapshot: MvccSnapshot[applied={T|T < 7315447064295251968}]
I20260806 06:30:18.436939 16059 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56136:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 06:30:18.437028 16059 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56136:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 06:30:18.437691 16059 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bbb2a969c88948d2848d8e7758b933c5: Sending DeleteTablet for 3 replicas of tablet 1db967605c1c4e30ad1d30533effb17a
I20260806 06:30:18.438005 16418 tablet_service.cc:1514] Processing DeleteTablet for tablet 1db967605c1c4e30ad1d30533effb17a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 06:30:18 UTC) from {username='kudu', principal='kudu/127.21.12.190@KRBTEST.COM'} at 127.0.0.1:40392
I20260806 06:30:18.438064 16225 tablet_service.cc:1514] Processing DeleteTablet for tablet 1db967605c1c4e30ad1d30533effb17a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 06:30:18 UTC) from {username='kudu', principal='kudu/127.21.12.190@KRBTEST.COM'} at 127.0.0.1:42096
I20260806 06:30:18.438194 16606 tablet_service.cc:1514] Processing DeleteTablet for tablet 1db967605c1c4e30ad1d30533effb17a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 06:30:18 UTC) from {username='kudu', principal='kudu/127.21.12.190@KRBTEST.COM'} at 127.0.0.1:56524
I20260806 06:30:18.438221 16751 tablet_replica.cc:330] T 1db967605c1c4e30ad1d30533effb17a P 3cd8c3ccfba54fd890e5266c7c062880: stopping tablet replica
I20260806 06:30:18.438315 16753 tablet_replica.cc:330] T 1db967605c1c4e30ad1d30533effb17a P 3ccdb3a132f74d1cba733cfec08f5583: stopping tablet replica
I20260806 06:30:18.438414 16753 raft_consensus.cc:2229] T 1db967605c1c4e30ad1d30533effb17a P 3ccdb3a132f74d1cba733cfec08f5583 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 06:30:18.438524 16753 raft_consensus.cc:2258] T 1db967605c1c4e30ad1d30533effb17a P 3ccdb3a132f74d1cba733cfec08f5583 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 06:30:18.438534 16751 raft_consensus.cc:2229] T 1db967605c1c4e30ad1d30533effb17a P 3cd8c3ccfba54fd890e5266c7c062880 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 06:30:18.438592 21554 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHhLhT/build/release/bin/kudu with pid 16141
I20260806 06:30:18.438731 16751 raft_consensus.cc:2258] T 1db967605c1c4e30ad1d30533effb17a P 3cd8c3ccfba54fd890e5266c7c062880 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 06:30:18.439353 16751 ts_tablet_manager.cc:1905] T 1db967605c1c4e30ad1d30533effb17a P 3cd8c3ccfba54fd890e5266c7c062880: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 06:30:18.441035 16751 ts_tablet_manager.cc:1918] T 1db967605c1c4e30ad1d30533effb17a P 3cd8c3ccfba54fd890e5266c7c062880: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 06:30:18.441170 16751 log.cc:1196] T 1db967605c1c4e30ad1d30533effb17a P 3cd8c3ccfba54fd890e5266c7c062880: Deleting WAL directory at /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785997581049705-21554-0/minicluster-data/ts-1/wal/wals/1db967605c1c4e30ad1d30533effb17a
I20260806 06:30:18.441530 16751 ts_tablet_manager.cc:1939] T 1db967605c1c4e30ad1d30533effb17a P 3cd8c3ccfba54fd890e5266c7c062880: Deleting consensus metadata
I20260806 06:30:18.442035 16040 catalog_manager.cc:4841] TS 3cd8c3ccfba54fd890e5266c7c062880 (127.21.12.130:41109): tablet 1db967605c1c4e30ad1d30533effb17a (table test-table [id=89c8f36547524628994a99564e3b0938]) successfully deleted
I20260806 06:30:18.442399 16755 tablet_replica.cc:330] T 1db967605c1c4e30ad1d30533effb17a P 57dacd8d2f9844388116c58f60d8226b: stopping tablet replica
I20260806 06:30:18.442548 16755 raft_consensus.cc:2229] T 1db967605c1c4e30ad1d30533effb17a P 57dacd8d2f9844388116c58f60d8226b [term 1 LEADER]: Raft consensus shutting down.
I20260806 06:30:18.442718 16755 raft_consensus.cc:2258] T 1db967605c1c4e30ad1d30533effb17a P 57dacd8d2f9844388116c58f60d8226b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 06:30:18.443612 16755 ts_tablet_manager.cc:1905] T 1db967605c1c4e30ad1d30533effb17a P 57dacd8d2f9844388116c58f60d8226b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 06:30:18.445189 16755 ts_tablet_manager.cc:1918] T 1db967605c1c4e30ad1d30533effb17a P 57dacd8d2f9844388116c58f60d8226b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 06:30:18.445330 16755 log.cc:1196] T 1db967605c1c4e30ad1d30533effb17a P 57dacd8d2f9844388116c58f60d8226b: Deleting WAL directory at /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785997581049705-21554-0/minicluster-data/ts-2/wal/wals/1db967605c1c4e30ad1d30533effb17a
I20260806 06:30:18.445672 16755 ts_tablet_manager.cc:1939] T 1db967605c1c4e30ad1d30533effb17a P 57dacd8d2f9844388116c58f60d8226b: Deleting consensus metadata
I20260806 06:30:18.446146 16035 catalog_manager.cc:4841] TS 57dacd8d2f9844388116c58f60d8226b (127.21.12.131:44469): tablet 1db967605c1c4e30ad1d30533effb17a (table test-table [id=89c8f36547524628994a99564e3b0938]) successfully deleted
W20260806 06:30:18.450388 16035 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.12.129:33727 (error 108)
I20260806 06:30:18.450759 21554 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHhLhT/build/release/bin/kudu with pid 16326
I20260806 06:30:18.453299 16705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:18.450827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.129:33727 (local address 127.0.0.1:42108)
0806 06:30:18.453215 (+ 2388us) negotiation.cc:107] Waiting for socket to connect
0806 06:30:18.453266 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.129:33727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2363}
W20260806 06:30:18.458009 16035 catalog_manager.cc:4584] TS 3ccdb3a132f74d1cba733cfec08f5583 (127.21.12.129:33727): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1db967605c1c4e30ad1d30533effb17a: Network error: Client connection negotiation failed: client connection to 127.21.12.129:33727: connect: Connection refused (error 111)
I20260806 06:30:18.466930 21554 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHhLhT/build/release/bin/kudu with pid 16520
I20260806 06:30:18.479043 21554 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHhLhT/build/release/bin/kudu with pid 16003
2026-08-06T06:30:18Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (1971 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-08-06T06:30:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T06:30:18Z Disabled control of system clock
I20260806 06:30:18.510160 21554 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHhLhT/build/release/bin/kudu
/tmp/dist-test-taskFHhLhT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785997581049705-21554-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785997581049705-21554-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785997581049705-21554-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785997581049705-21554-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.12.190:35715
--webserver_interface=127.21.12.190
--webserver_port=0
--builtin_ntp_servers=127.21.12.148:45789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.12.190:35715
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260806 06:30:18.590015 16760 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 06:30:18.590226 16760 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 06:30:18.590245 16760 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 06:30:18.591388 16760 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 06:30:18.591436 16760 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 06:30:18.591449 16760 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 06:30:18.591460 16760 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 06:30:18.591470 16760 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 06:30:18.592796 16760 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.148:45789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785997581049705-21554-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785997581049705-21554-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.12.190:35715
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.12.190:35715
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785997581049705-21554-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.12.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16760
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785997581049705-21554-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 05:11:02 UTC on ccycloud.quasar-hvzbbz.root.comops.site
build id 4569617
I20260806 06:30:18.593019 16760 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 06:30:18.593282 16760 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 06:30:18.597846 16768 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:30:18.597810 16767 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:30:18.597820 16770 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:30:18.598091 16760 server_base.cc:1034] running on GCE node
I20260806 06:30:18.598285 16760 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 06:30:18.598608 16760 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 06:30:18.599745 16760 hybrid_clock.cc:648] HybridClock initialized: now 1785997818599732 us; error 28 us; skew 500 ppm
I20260806 06:30:18.600850 16760 webserver.cc:460] Webserver started at http://127.21.12.190:41389/ using document root <none> and password file <none>
I20260806 06:30:18.601030 16760 fs_manager.cc:359] Metadata directory not provided
I20260806 06:30:18.601068 16760 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 06:30:18.601161 16760 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 06:30:18.602146 16760 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785997581049705-21554-0/minicluster-data/master-0/data/instance:
uuid: "996b5cec09f84daaba3aa73b9e82aade"
format_stamp: "Formatted at 2026-08-06 06:30:18 on dist-test-slave-4tdj"
server_key: "adfecbdfe8e1ee06341ab30872f691b1"
server_key_iv: "01caca663347fb1d516f6db1ec0691b5"
server_key_version: "encryptionkey@0"
I20260806 06:30:18.602478 16760 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785997581049705-21554-0/minicluster-data/master-0/wal/instance:
uuid: "996b5cec09f84daaba3aa73b9e82aade"
format_stamp: "Formatted at 2026-08-06 06:30:18 on dist-test-slave-4tdj"
server_key: "adfecbdfe8e1ee06341ab30872f691b1"
server_key_iv: "01caca663347fb1d516f6db1ec0691b5"
server_key_version: "encryptionkey@0"
I20260806 06:30:18.603667 16760 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260806 06:30:18.604387 16780 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 06:30:18.604594 16760 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260806 06:30:18.604676 16760 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785997581049705-21554-0/minicluster-data/master-0/data,/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785997581049705-21554-0/minicluster-data/master-0/wal
uuid: "996b5cec09f84daaba3aa73b9e82aade"
format_stamp: "Formatted at 2026-08-06 06:30:18 on dist-test-slave-4tdj"
server_key: "adfecbdfe8e1ee06341ab30872f691b1"
server_key_iv: "01caca663347fb1d516f6db1ec0691b5"
server_key_version: "encryptionkey@0"
I20260806 06:30:18.604732 16760 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785997581049705-21554-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785997581049705-21554-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785997581049705-21554-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 06:30:18.620988 16760 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 06:30:18.621284 16760 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 06:30:18.621402 16760 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 06:30:18.625571 16760 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.190:35715
I20260806 06:30:18.625613 16854 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.190:35715 every 8 connection(s)
I20260806 06:30:18.625945 16760 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785997581049705-21554-0/minicluster-data/master-0/data/info.pb
I20260806 06:30:18.627008 16855 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 06:30:18.629385 16855 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 996b5cec09f84daaba3aa73b9e82aade: Bootstrap starting.
I20260806 06:30:18.629968 16855 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 996b5cec09f84daaba3aa73b9e82aade: Neither blocks nor log segments found. Creating new log.
I20260806 06:30:18.630242 16855 log.cc:824] T 00000000000000000000000000000000 P 996b5cec09f84daaba3aa73b9e82aade: Log is configured to *not* fsync() on all Append() calls
I20260806 06:30:18.630926 16855 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 996b5cec09f84daaba3aa73b9e82aade: No bootstrap required, opened a new log
I20260806 06:30:18.632539 16855 raft_consensus.cc:348] T 00000000000000000000000000000000 P 996b5cec09f84daaba3aa73b9e82aade [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "996b5cec09f84daaba3aa73b9e82aade" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 35715 } }
I20260806 06:30:18.632675 16855 raft_consensus.cc:374] T 00000000000000000000000000000000 P 996b5cec09f84daaba3aa73b9e82aade [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 06:30:18.632707 16855 raft_consensus.cc:729] T 00000000000000000000000000000000 P 996b5cec09f84daaba3aa73b9e82aade [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 996b5cec09f84daaba3aa73b9e82aade, State: Initialized, Role: FOLLOWER
I20260806 06:30:18.632839 16855 consensus_queue.cc:260] T 00000000000000000000000000000000 P 996b5cec09f84daaba3aa73b9e82aade [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "996b5cec09f84daaba3aa73b9e82aade" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 35715 } }
I20260806 06:30:18.632922 16855 raft_consensus.cc:388] T 00000000000000000000000000000000 P 996b5cec09f84daaba3aa73b9e82aade [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 06:30:18.632959 16855 raft_consensus.cc:482] T 00000000000000000000000000000000 P 996b5cec09f84daaba3aa73b9e82aade [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 06:30:18.633006 16855 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 996b5cec09f84daaba3aa73b9e82aade [term 0 FOLLOWER]: Advancing to term 1
I20260806 06:30:18.633502 16855 raft_consensus.cc:504] T 00000000000000000000000000000000 P 996b5cec09f84daaba3aa73b9e82aade [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "996b5cec09f84daaba3aa73b9e82aade" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 35715 } }
I20260806 06:30:18.633595 16855 leader_election.cc:302] T 00000000000000000000000000000000 P 996b5cec09f84daaba3aa73b9e82aade [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 996b5cec09f84daaba3aa73b9e82aade; no voters:
I20260806 06:30:18.633751 16855 leader_election.cc:288] T 00000000000000000000000000000000 P 996b5cec09f84daaba3aa73b9e82aade [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 06:30:18.633800 16859 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 996b5cec09f84daaba3aa73b9e82aade [term 1 FOLLOWER]: Leader election won for term 1
I20260806 06:30:18.633936 16859 raft_consensus.cc:686] T 00000000000000000000000000000000 P 996b5cec09f84daaba3aa73b9e82aade [term 1 LEADER]: Becoming Leader. State: Replica: 996b5cec09f84daaba3aa73b9e82aade, State: Running, Role: LEADER
I20260806 06:30:18.634018 16855 sys_catalog.cc:564] T 00000000000000000000000000000000 P 996b5cec09f84daaba3aa73b9e82aade [sys.catalog]: configured and running, proceeding with master startup.
I20260806 06:30:18.634128 16859 consensus_queue.cc:237] T 00000000000000000000000000000000 P 996b5cec09f84daaba3aa73b9e82aade [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "996b5cec09f84daaba3aa73b9e82aade" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 35715 } }
I20260806 06:30:18.634498 21554 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHhLhT/build/release/bin/kudu as pid 16760
I20260806 06:30:18.634614 21554 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785997581049705-21554-0/minicluster-data/master-0/wal/instance
I20260806 06:30:18.634606 16862 sys_catalog.cc:455] T 00000000000000000000000000000000 P 996b5cec09f84daaba3aa73b9e82aade [sys.catalog]: SysCatalogTable state changed. Reason: New leader 996b5cec09f84daaba3aa73b9e82aade. Latest consensus state: current_term: 1 leader_uuid: "996b5cec09f84daaba3aa73b9e82aade" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "996b5cec09f84daaba3aa73b9e82aade" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 35715 } } }
I20260806 06:30:18.634680 16862 sys_catalog.cc:458] T 00000000000000000000000000000000 P 996b5cec09f84daaba3aa73b9e82aade [sys.catalog]: This master's current role is: LEADER
I20260806 06:30:18.634825 21554 external_mini_cluster.cc:1428] Setting key 87d4e1f5c2cbc42c1e30992258dcbb9b
I20260806 06:30:18.634824 16861 sys_catalog.cc:455] T 00000000000000000000000000000000 P 996b5cec09f84daaba3aa73b9e82aade [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "996b5cec09f84daaba3aa73b9e82aade" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "996b5cec09f84daaba3aa73b9e82aade" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 35715 } } }
I20260806 06:30:18.635079 16861 sys_catalog.cc:458] T 00000000000000000000000000000000 P 996b5cec09f84daaba3aa73b9e82aade [sys.catalog]: This master's current role is: LEADER
I20260806 06:30:18.635087 16874 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 06:30:18.635668 16874 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 06:30:18.637333 16874 catalog_manager.cc:1347] Generated new cluster ID: 1eae7701fd594d6a952ffdc642c7915c
I20260806 06:30:18.637383 16874 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 06:30:18.640600 16886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:18.635289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59534 (local address 127.21.12.190:35715)
0806 06:30:18.635550 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0806 06:30:18.635560 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 06:30:18.635595 (+ 35us) server_negotiation.cc:407] Connection header received
0806 06:30:18.635711 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:30:18.635719 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:30:18.635838 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:30:18.636199 (+ 361us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:30:18.636281 (+ 82us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:18.637415 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:18.638502 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:18.639095 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:18.639199 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:30:18.639858 (+ 659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:30:18.639862 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:30:18.639862 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 06:30:18.640058 (+ 196us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:30:18.640097 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 06:30:18.640110 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:30:18.640168 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0806 06:30:18.640244 (+ 76us) server_negotiation.cc:299] Negotiation successful
0806 06:30:18.640399 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":67,"threads_started":1}
I20260806 06:30:18.657128 16874 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 06:30:18.657645 16874 catalog_manager.cc:1504] Loading token signing keys...
I20260806 06:30:18.662873 16874 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 996b5cec09f84daaba3aa73b9e82aade: Generated new TSK 0
I20260806 06:30:18.663089 16874 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 06:30:18.668656 21554 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHhLhT/build/release/bin/kudu
/tmp/dist-test-taskFHhLhT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785997581049705-21554-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785997581049705-21554-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785997581049705-21554-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785997581049705-21554-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.12.129:0
--local_ip_for_outbound_sockets=127.21.12.129
--webserver_interface=127.21.12.129
--webserver_port=0
--tserver_master_addrs=127.21.12.190:35715
--builtin_ntp_servers=127.21.12.148:45789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260806 06:30:18.756935 16893 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 06:30:18.757141 16893 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 06:30:18.757161 16893 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 06:30:18.758275 16893 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 06:30:18.758375 16893 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 06:30:18.758404 16893 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.12.129
I20260806 06:30:18.759742 16893 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.148:45789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785997581049705-21554-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785997581049705-21554-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.12.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785997581049705-21554-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.12.129
--webserver_port=0
--tserver_master_addrs=127.21.12.190:35715
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16893
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.12.129
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785997581049705-21554-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 05:11:02 UTC on ccycloud.quasar-hvzbbz.root.comops.site
build id 4569617
I20260806 06:30:18.759958 16893 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 06:30:18.760233 16893 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 06:30:18.764935 16900 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:30:18.764902 16905 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:30:18.764917 16901 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:30:18.765180 16893 server_base.cc:1034] running on GCE node
I20260806 06:30:18.765352 16893 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 06:30:18.765583 16893 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 06:30:18.766732 16893 hybrid_clock.cc:648] HybridClock initialized: now 1785997818766718 us; error 29 us; skew 500 ppm
I20260806 06:30:18.767836 16893 webserver.cc:460] Webserver started at http://127.21.12.129:33501/ using document root <none> and password file <none>
I20260806 06:30:18.768041 16893 fs_manager.cc:359] Metadata directory not provided
I20260806 06:30:18.768085 16893 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 06:30:18.768206 16893 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 06:30:18.769224 16893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785997581049705-21554-0/minicluster-data/ts-0/data/instance:
uuid: "c5ad55bc1785499d854288a27769fefe"
format_stamp: "Formatted at 2026-08-06 06:30:18 on dist-test-slave-4tdj"
server_key: "5a410ef8d686d0a6a427cf716a3b6e98"
server_key_iv: "67a2285a0fd46e277240e0d7dcb8d35e"
server_key_version: "encryptionkey@0"
I20260806 06:30:18.769547 16893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785997581049705-21554-0/minicluster-data/ts-0/wal/instance:
uuid: "c5ad55bc1785499d854288a27769fefe"
format_stamp: "Formatted at 2026-08-06 06:30:18 on dist-test-slave-4tdj"
server_key: "5a410ef8d686d0a6a427cf716a3b6e98"
server_key_iv: "67a2285a0fd46e277240e0d7dcb8d35e"
server_key_version: "encryptionkey@0"
I20260806 06:30:18.770851 16893 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260806 06:30:18.771534 16914 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 06:30:18.771683 16893 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260806 06:30:18.771765 16893 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785997581049705-21554-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785997581049705-21554-0/minicluster-data/ts-0/wal
uuid: "c5ad55bc1785499d854288a27769fefe"
format_stamp: "Formatted at 2026-08-06 06:30:18 on dist-test-slave-4tdj"
server_key: "5a410ef8d686d0a6a427cf716a3b6e98"
server_key_iv: "67a2285a0fd46e277240e0d7dcb8d35e"
server_key_version: "encryptionkey@0"
I20260806 06:30:18.771827 16893 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785997581049705-21554-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785997581049705-21554-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785997581049705-21554-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 06:30:18.787647 16893 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 06:30:18.787910 16893 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 06:30:18.788028 16893 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 06:30:18.788231 16893 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 06:30:18.788548 16893 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 06:30:18.788591 16893 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 06:30:18.788620 16893 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 06:30:18.788645 16893 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 06:30:18.793988 16893 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.129:33641
I20260806 06:30:18.794067 17080 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.129:33641 every 8 connection(s)
I20260806 06:30:18.794384 16893 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785997581049705-21554-0/minicluster-data/ts-0/data/info.pb
I20260806 06:30:18.794831 21554 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHhLhT/build/release/bin/kudu as pid 16893
I20260806 06:30:18.794929 21554 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785997581049705-21554-0/minicluster-data/ts-0/wal/instance
I20260806 06:30:18.795115 21554 external_mini_cluster.cc:1428] Setting key 706b24d2fcacfa8c8e0de55b401144b2
I20260806 06:30:18.796058 21554 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHhLhT/build/release/bin/kudu
/tmp/dist-test-taskFHhLhT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785997581049705-21554-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785997581049705-21554-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785997581049705-21554-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785997581049705-21554-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.12.130:0
--local_ip_for_outbound_sockets=127.21.12.130
--webserver_interface=127.21.12.130
--webserver_port=0
--tserver_master_addrs=127.21.12.190:35715
--builtin_ntp_servers=127.21.12.148:45789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260806 06:30:18.799953 16886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:18.795613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.129:47115 (local address 127.21.12.190:35715)
0806 06:30:18.795662 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0806 06:30:18.795664 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:30:18.795931 (+ 267us) server_negotiation.cc:407] Connection header received
0806 06:30:18.796478 (+ 547us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:30:18.796479 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:30:18.796490 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:30:18.796541 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:30:18.797193 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:18.797689 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:18.798680 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:18.798882 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:18.798936 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:30:18.799693 (+ 757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:30:18.799694 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:30:18.799695 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 06:30:18.799702 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:30:18.799717 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 06:30:18.799721 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:30:18.799767 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0806 06:30:18.799843 (+ 76us) server_negotiation.cc:299] Negotiation successful
0806 06:30:18.799875 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260806 06:30:18.800148 17083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:18.795628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:35715 (local address 127.21.12.129:47115)
0806 06:30:18.795810 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0806 06:30:18.795848 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0806 06:30:18.796384 (+ 536us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:30:18.796607 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:30:18.796612 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:30:18.796622 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0806 06:30:18.797110 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:30:18.797115 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:30:18.797770 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:30:18.797771 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:30:18.798603 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:30:18.798606 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:30:18.798952 (+ 346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:30:18.798953 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:30:18.799081 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:30:18.799561 (+ 480us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 06:30:18.799561 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0806 06:30:18.799592 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 06:30:18.799614 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 06:30:18.799772 (+ 158us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 06:30:18.799778 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 06:30:18.799781 (+ 3us) client_negotiation.cc:769] Sending connection context
0806 06:30:18.799841 (+ 60us) client_negotiation.cc:240] Negotiation successful
0806 06:30:18.799939 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":88,"threads_started":1}
I20260806 06:30:18.800613 17081 heartbeater.cc:347] Connected to a master server at 127.21.12.190:35715
I20260806 06:30:18.800719 17081 heartbeater.cc:464] Registering TS with master...
I20260806 06:30:18.800915 17081 heartbeater.cc:511] Master 127.21.12.190:35715 requested a full tablet report, sending...
I20260806 06:30:18.801443 16806 ts_manager.cc:194] Registered new tserver with Master: c5ad55bc1785499d854288a27769fefe (127.21.12.129:33641)
I20260806 06:30:18.802078 16806 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.12.129:47115
W20260806 06:30:18.877341 17084 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 06:30:18.877552 17084 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 06:30:18.877573 17084 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 06:30:18.878747 17084 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 06:30:18.878788 17084 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 06:30:18.878814 17084 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.12.130
I20260806 06:30:18.880168 17084 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.148:45789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785997581049705-21554-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785997581049705-21554-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.12.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785997581049705-21554-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.12.130
--webserver_port=0
--tserver_master_addrs=127.21.12.190:35715
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17084
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.12.130
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785997581049705-21554-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 05:11:02 UTC on ccycloud.quasar-hvzbbz.root.comops.site
build id 4569617
I20260806 06:30:18.880393 17084 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 06:30:18.880707 17084 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 06:30:18.885417 17096 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:30:18.885439 17093 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:30:18.885493 17091 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:30:18.885571 17084 server_base.cc:1034] running on GCE node
I20260806 06:30:18.885839 17084 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 06:30:18.886047 17084 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 06:30:18.887197 17084 hybrid_clock.cc:648] HybridClock initialized: now 1785997818887184 us; error 27 us; skew 500 ppm
I20260806 06:30:18.888319 17084 webserver.cc:460] Webserver started at http://127.21.12.130:40371/ using document root <none> and password file <none>
I20260806 06:30:18.888566 17084 fs_manager.cc:359] Metadata directory not provided
I20260806 06:30:18.888618 17084 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 06:30:18.888749 17084 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 06:30:18.889745 17084 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785997581049705-21554-0/minicluster-data/ts-1/data/instance:
uuid: "3b60d1e056b24cc49271bc406fac687f"
format_stamp: "Formatted at 2026-08-06 06:30:18 on dist-test-slave-4tdj"
server_key: "016de658534305aa72fde2e61a672a99"
server_key_iv: "611145c5136f64c753150b5d27defec9"
server_key_version: "encryptionkey@0"
I20260806 06:30:18.890074 17084 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785997581049705-21554-0/minicluster-data/ts-1/wal/instance:
uuid: "3b60d1e056b24cc49271bc406fac687f"
format_stamp: "Formatted at 2026-08-06 06:30:18 on dist-test-slave-4tdj"
server_key: "016de658534305aa72fde2e61a672a99"
server_key_iv: "611145c5136f64c753150b5d27defec9"
server_key_version: "encryptionkey@0"
I20260806 06:30:18.891386 17084 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260806 06:30:18.892040 17106 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 06:30:18.892199 17084 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260806 06:30:18.892282 17084 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785997581049705-21554-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785997581049705-21554-0/minicluster-data/ts-1/wal
uuid: "3b60d1e056b24cc49271bc406fac687f"
format_stamp: "Formatted at 2026-08-06 06:30:18 on dist-test-slave-4tdj"
server_key: "016de658534305aa72fde2e61a672a99"
server_key_iv: "611145c5136f64c753150b5d27defec9"
server_key_version: "encryptionkey@0"
I20260806 06:30:18.892343 17084 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785997581049705-21554-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785997581049705-21554-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785997581049705-21554-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 06:30:18.898427 17084 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 06:30:18.898666 17084 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 06:30:18.898779 17084 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 06:30:18.898955 17084 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 06:30:18.899236 17084 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 06:30:18.899269 17084 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 06:30:18.899295 17084 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 06:30:18.899319 17084 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 06:30:18.904646 17084 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.130:42101
I20260806 06:30:18.904706 17261 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.130:42101 every 8 connection(s)
I20260806 06:30:18.904981 17084 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785997581049705-21554-0/minicluster-data/ts-1/data/info.pb
I20260806 06:30:18.909577 16886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:18.905630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.130:53429 (local address 127.21.12.190:35715)
0806 06:30:18.905709 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0806 06:30:18.905711 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:30:18.905876 (+ 165us) server_negotiation.cc:407] Connection header received
0806 06:30:18.906359 (+ 483us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:30:18.906360 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:30:18.906374 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:30:18.906424 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:30:18.907057 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:18.907548 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:18.908459 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:18.908654 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:18.908696 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:30:18.909384 (+ 688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:30:18.909385 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:30:18.909385 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 06:30:18.909389 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:30:18.909398 (+ 9us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 06:30:18.909401 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:30:18.909432 (+ 31us) server_negotiation.cc:1035] Waiting for connection context
0806 06:30:18.909481 (+ 49us) server_negotiation.cc:299] Negotiation successful
0806 06:30:18.909503 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260806 06:30:18.909842 17266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:18.905604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:35715 (local address 127.21.12.130:53429)
0806 06:30:18.905772 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0806 06:30:18.905802 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0806 06:30:18.906241 (+ 439us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:30:18.906476 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:30:18.906480 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:30:18.906488 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0806 06:30:18.906988 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:30:18.906993 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:30:18.907615 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:30:18.907616 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:30:18.908397 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:30:18.908399 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:30:18.908714 (+ 315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:30:18.908714 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:30:18.908821 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:30:18.909277 (+ 456us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 06:30:18.909277 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0806 06:30:18.909304 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 06:30:18.909326 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 06:30:18.909434 (+ 108us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 06:30:18.909436 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 06:30:18.909438 (+ 2us) client_negotiation.cc:769] Sending connection context
0806 06:30:18.909481 (+ 43us) client_negotiation.cc:240] Negotiation successful
0806 06:30:18.909550 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":74,"threads_started":1}
I20260806 06:30:18.910372 17262 heartbeater.cc:347] Connected to a master server at 127.21.12.190:35715
I20260806 06:30:18.910470 17262 heartbeater.cc:464] Registering TS with master...
I20260806 06:30:18.910641 17262 heartbeater.cc:511] Master 127.21.12.190:35715 requested a full tablet report, sending...
I20260806 06:30:18.910966 16806 ts_manager.cc:194] Registered new tserver with Master: 3b60d1e056b24cc49271bc406fac687f (127.21.12.130:42101)
I20260806 06:30:18.911096 21554 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHhLhT/build/release/bin/kudu as pid 17084
I20260806 06:30:18.911196 21554 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785997581049705-21554-0/minicluster-data/ts-1/wal/instance
I20260806 06:30:18.911375 21554 external_mini_cluster.cc:1428] Setting key 2b47cc7279692f8058d7c8cc304d00b3
I20260806 06:30:18.911396 16806 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.12.130:53429
I20260806 06:30:18.912248 21554 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHhLhT/build/release/bin/kudu
/tmp/dist-test-taskFHhLhT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785997581049705-21554-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785997581049705-21554-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785997581049705-21554-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785997581049705-21554-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.12.131:0
--local_ip_for_outbound_sockets=127.21.12.131
--webserver_interface=127.21.12.131
--webserver_port=0
--tserver_master_addrs=127.21.12.190:35715
--builtin_ntp_servers=127.21.12.148:45789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260806 06:30:18.991663 17268 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 06:30:18.991863 17268 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 06:30:18.991891 17268 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 06:30:18.992996 17268 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 06:30:18.993034 17268 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 06:30:18.993074 17268 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.12.131
I20260806 06:30:18.994398 17268 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.148:45789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785997581049705-21554-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785997581049705-21554-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.12.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785997581049705-21554-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.12.131
--webserver_port=0
--tserver_master_addrs=127.21.12.190:35715
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17268
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.12.131
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785997581049705-21554-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 05:11:02 UTC on ccycloud.quasar-hvzbbz.root.comops.site
build id 4569617
I20260806 06:30:18.994650 17268 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 06:30:18.994932 17268 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 06:30:18.999595 17280 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:30:18.999681 17276 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:30:18.999675 17275 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:30:18.999771 17268 server_base.cc:1034] running on GCE node
I20260806 06:30:19.000078 17268 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 06:30:19.000306 17268 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 06:30:19.001451 17268 hybrid_clock.cc:648] HybridClock initialized: now 1785997819001432 us; error 36 us; skew 500 ppm
I20260806 06:30:19.002616 17268 webserver.cc:460] Webserver started at http://127.21.12.131:41281/ using document root <none> and password file <none>
I20260806 06:30:19.002817 17268 fs_manager.cc:359] Metadata directory not provided
I20260806 06:30:19.002856 17268 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 06:30:19.002948 17268 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 06:30:19.003932 17268 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785997581049705-21554-0/minicluster-data/ts-2/data/instance:
uuid: "07f80cff67044cb38d19f026639995e7"
format_stamp: "Formatted at 2026-08-06 06:30:18 on dist-test-slave-4tdj"
server_key: "1d16fbcc2c9ce453217e416b5e5afb88"
server_key_iv: "dccf8d0357ec0f044e00d8024b23d241"
server_key_version: "encryptionkey@0"
I20260806 06:30:19.004228 17268 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785997581049705-21554-0/minicluster-data/ts-2/wal/instance:
uuid: "07f80cff67044cb38d19f026639995e7"
format_stamp: "Formatted at 2026-08-06 06:30:18 on dist-test-slave-4tdj"
server_key: "1d16fbcc2c9ce453217e416b5e5afb88"
server_key_iv: "dccf8d0357ec0f044e00d8024b23d241"
server_key_version: "encryptionkey@0"
I20260806 06:30:19.005370 17268 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260806 06:30:19.005972 17291 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 06:30:19.006121 17268 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260806 06:30:19.006203 17268 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785997581049705-21554-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785997581049705-21554-0/minicluster-data/ts-2/wal
uuid: "07f80cff67044cb38d19f026639995e7"
format_stamp: "Formatted at 2026-08-06 06:30:18 on dist-test-slave-4tdj"
server_key: "1d16fbcc2c9ce453217e416b5e5afb88"
server_key_iv: "dccf8d0357ec0f044e00d8024b23d241"
server_key_version: "encryptionkey@0"
I20260806 06:30:19.006258 17268 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785997581049705-21554-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785997581049705-21554-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785997581049705-21554-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 06:30:19.024416 17268 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 06:30:19.024755 17268 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 06:30:19.024914 17268 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 06:30:19.025153 17268 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 06:30:19.025534 17268 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 06:30:19.025589 17268 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 06:30:19.025627 17268 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 06:30:19.025650 17268 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 06:30:19.031589 17268 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.131:36973
I20260806 06:30:19.031649 17459 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.131:36973 every 8 connection(s)
I20260806 06:30:19.031939 17268 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785997581049705-21554-0/minicluster-data/ts-2/data/info.pb
I20260806 06:30:19.036856 16886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:19.032606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.131:43981 (local address 127.21.12.190:35715)
0806 06:30:19.032668 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0806 06:30:19.032670 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:30:19.032902 (+ 232us) server_negotiation.cc:407] Connection header received
0806 06:30:19.033406 (+ 504us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:30:19.033407 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:30:19.033420 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:30:19.033472 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:30:19.034148 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:19.034694 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:19.035651 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:19.035851 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:19.035905 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:30:19.036602 (+ 697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:30:19.036604 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:30:19.036604 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 06:30:19.036608 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:30:19.036625 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 06:30:19.036629 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:30:19.036675 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0806 06:30:19.036747 (+ 72us) server_negotiation.cc:299] Negotiation successful
0806 06:30:19.036776 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260806 06:30:19.037029 17465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:19.032610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:35715 (local address 127.21.12.131:43981)
0806 06:30:19.032788 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0806 06:30:19.032822 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0806 06:30:19.033305 (+ 483us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:30:19.033540 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:30:19.033544 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:30:19.033553 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0806 06:30:19.034063 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:30:19.034068 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:30:19.034778 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:30:19.034779 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:30:19.035570 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:30:19.035573 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:30:19.035915 (+ 342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:30:19.035916 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:30:19.036035 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:30:19.036476 (+ 441us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 06:30:19.036477 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0806 06:30:19.036503 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 06:30:19.036524 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 06:30:19.036680 (+ 156us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 06:30:19.036686 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 06:30:19.036689 (+ 3us) client_negotiation.cc:769] Sending connection context
0806 06:30:19.036745 (+ 56us) client_negotiation.cc:240] Negotiation successful
0806 06:30:19.036842 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":79,"threads_started":1}
I20260806 06:30:19.037358 21554 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHhLhT/build/release/bin/kudu as pid 17268
I20260806 06:30:19.037453 21554 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785997581049705-21554-0/minicluster-data/ts-2/wal/instance
I20260806 06:30:19.037575 17461 heartbeater.cc:347] Connected to a master server at 127.21.12.190:35715
I20260806 06:30:19.037638 21554 external_mini_cluster.cc:1428] Setting key 373cd1e606b6ce790b546b417470d1a2
I20260806 06:30:19.037662 17461 heartbeater.cc:464] Registering TS with master...
I20260806 06:30:19.037827 17461 heartbeater.cc:511] Master 127.21.12.190:35715 requested a full tablet report, sending...
I20260806 06:30:19.038189 16806 ts_manager.cc:194] Registered new tserver with Master: 07f80cff67044cb38d19f026639995e7 (127.21.12.131:36973)
I20260806 06:30:19.038645 16806 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.12.131:43981
I20260806 06:30:19.038682 21554 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 06:30:19.042681 16886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:19.040014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59538 (local address 127.21.12.190:35715)
0806 06:30:19.040075 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0806 06:30:19.040077 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:30:19.040173 (+ 96us) server_negotiation.cc:407] Connection header received
0806 06:30:19.040236 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:30:19.040237 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:30:19.040246 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:30:19.040290 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:30:19.040422 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:19.040868 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:19.041588 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:19.041778 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:19.041829 (+ 51us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:30:19.042423 (+ 594us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:30:19.042425 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:30:19.042425 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 06:30:19.042429 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:30:19.042443 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 06:30:19.042447 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:30:19.042527 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0806 06:30:19.042597 (+ 70us) server_negotiation.cc:299] Negotiation successful
0806 06:30:19.042606 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260806 06:30:19.044142 16806 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59538:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 06:30:19.045704 16806 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 06:30:19.051741 17477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:19.047612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.130:42101 (local address 127.0.0.1:57346)
0806 06:30:19.047770 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0806 06:30:19.047805 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0806 06:30:19.047870 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:30:19.048197 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:30:19.048198 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:30:19.048200 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 06:30:19.048255 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:30:19.048257 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:30:19.049479 (+ 1222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:30:19.049480 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:30:19.050544 (+ 1064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:30:19.050546 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:30:19.051455 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:30:19.051456 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:30:19.051542 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:30:19.051555 (+ 13us) client_negotiation.cc:769] Sending connection context
0806 06:30:19.051619 (+ 64us) client_negotiation.cc:240] Negotiation successful
0806 06:30:19.051667 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"mutex_wait_us":3,"thread_start_us":57,"threads_started":1}
I20260806 06:30:19.051751 17480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:19.047838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57346 (local address 127.21.12.130:42101)
0806 06:30:19.048035 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0806 06:30:19.048039 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 06:30:19.048047 (+ 8us) server_negotiation.cc:407] Connection header received
0806 06:30:19.048077 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:30:19.048079 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:30:19.048090 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:30:19.048197 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 06:30:19.048318 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:19.049395 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:19.050644 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:19.051355 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:19.051435 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:30:19.051528 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0806 06:30:19.051633 (+ 105us) server_negotiation.cc:299] Negotiation successful
0806 06:30:19.051671 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":84,"threads_started":1}
I20260806 06:30:19.051891 17478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:19.047689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36892 (local address 127.21.12.129:33641)
0806 06:30:19.048126 (+ 437us) server_negotiation.cc:206] Beginning negotiation
0806 06:30:19.048130 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 06:30:19.048138 (+ 8us) server_negotiation.cc:407] Connection header received
0806 06:30:19.048173 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:30:19.048175 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:30:19.048189 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:30:19.048372 (+ 183us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 06:30:19.048415 (+ 43us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:19.049203 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:19.050644 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:19.051390 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:19.051445 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:30:19.051528 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0806 06:30:19.051780 (+ 252us) server_negotiation.cc:299] Negotiation successful
0806 06:30:19.051819 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":395,"thread_start_us":74,"threads_started":1}
I20260806 06:30:19.052233 17474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:19.047613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.131:36973 (local address 127.0.0.1:34540)
0806 06:30:19.047854 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0806 06:30:19.047865 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0806 06:30:19.047920 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:30:19.048144 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:30:19.048147 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:30:19.048150 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 06:30:19.048209 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:30:19.048211 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:30:19.050361 (+ 2150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:30:19.050362 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:30:19.051201 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:30:19.051204 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:30:19.052031 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:30:19.052033 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:30:19.052080 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:30:19.052083 (+ 3us) client_negotiation.cc:769] Sending connection context
0806 06:30:19.052130 (+ 47us) client_negotiation.cc:240] Negotiation successful
0806 06:30:19.052164 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":85,"threads_started":1}
I20260806 06:30:19.052250 17479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:19.047807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34540 (local address 127.21.12.131:36973)
0806 06:30:19.047967 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0806 06:30:19.047971 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 06:30:19.047980 (+ 9us) server_negotiation.cc:407] Connection header received
0806 06:30:19.048016 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:30:19.048018 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:30:19.048032 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:30:19.048126 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 06:30:19.048288 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:19.049394 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:19.051277 (+ 1883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:19.051940 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:19.052012 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:30:19.052088 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0806 06:30:19.052143 (+ 55us) server_negotiation.cc:299] Negotiation successful
0806 06:30:19.052176 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":75,"threads_started":1}
I20260806 06:30:19.052431 17475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:19.047613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.129:33641 (local address 127.0.0.1:36892)
0806 06:30:19.047969 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0806 06:30:19.047977 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0806 06:30:19.048013 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:30:19.048277 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:30:19.048278 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:30:19.048279 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 06:30:19.048313 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:30:19.048314 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:30:19.049274 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:30:19.049275 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:30:19.050543 (+ 1268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:30:19.050546 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:30:19.051565 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:30:19.051566 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:30:19.051629 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:30:19.051633 (+ 4us) client_negotiation.cc:769] Sending connection context
0806 06:30:19.051678 (+ 45us) client_negotiation.cc:240] Negotiation successful
0806 06:30:19.051718 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":342,"mutex_wait_us":3,"thread_start_us":79,"threads_started":1}
I20260806 06:30:19.052908 17177 tablet_service.cc:1467] Processing CreateTablet for tablet ab6d3ecf31dd4645b797be5cf970a6ec (DEFAULT_TABLE table=test-table [id=759c450ecadc4c5d82678cedf409ce47]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 06:30:19.052960 17356 tablet_service.cc:1467] Processing CreateTablet for tablet ab6d3ecf31dd4645b797be5cf970a6ec (DEFAULT_TABLE table=test-table [id=759c450ecadc4c5d82678cedf409ce47]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 06:30:19.053069 16986 tablet_service.cc:1467] Processing CreateTablet for tablet ab6d3ecf31dd4645b797be5cf970a6ec (DEFAULT_TABLE table=test-table [id=759c450ecadc4c5d82678cedf409ce47]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 06:30:19.053288 17177 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab6d3ecf31dd4645b797be5cf970a6ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 06:30:19.053297 17356 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab6d3ecf31dd4645b797be5cf970a6ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 06:30:19.053357 16986 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab6d3ecf31dd4645b797be5cf970a6ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 06:30:19.055646 17482 tablet_bootstrap.cc:492] T ab6d3ecf31dd4645b797be5cf970a6ec P 07f80cff67044cb38d19f026639995e7: Bootstrap starting.
I20260806 06:30:19.056463 17483 tablet_bootstrap.cc:492] T ab6d3ecf31dd4645b797be5cf970a6ec P 3b60d1e056b24cc49271bc406fac687f: Bootstrap starting.
I20260806 06:30:19.056595 17487 tablet_bootstrap.cc:492] T ab6d3ecf31dd4645b797be5cf970a6ec P c5ad55bc1785499d854288a27769fefe: Bootstrap starting.
I20260806 06:30:19.056864 17482 tablet_bootstrap.cc:654] T ab6d3ecf31dd4645b797be5cf970a6ec P 07f80cff67044cb38d19f026639995e7: Neither blocks nor log segments found. Creating new log.
I20260806 06:30:19.057137 17483 tablet_bootstrap.cc:654] T ab6d3ecf31dd4645b797be5cf970a6ec P 3b60d1e056b24cc49271bc406fac687f: Neither blocks nor log segments found. Creating new log.
I20260806 06:30:19.057150 17487 tablet_bootstrap.cc:654] T ab6d3ecf31dd4645b797be5cf970a6ec P c5ad55bc1785499d854288a27769fefe: Neither blocks nor log segments found. Creating new log.
I20260806 06:30:19.057183 17482 log.cc:824] T ab6d3ecf31dd4645b797be5cf970a6ec P 07f80cff67044cb38d19f026639995e7: Log is configured to *not* fsync() on all Append() calls
I20260806 06:30:19.057416 17487 log.cc:824] T ab6d3ecf31dd4645b797be5cf970a6ec P c5ad55bc1785499d854288a27769fefe: Log is configured to *not* fsync() on all Append() calls
I20260806 06:30:19.057447 17483 log.cc:824] T ab6d3ecf31dd4645b797be5cf970a6ec P 3b60d1e056b24cc49271bc406fac687f: Log is configured to *not* fsync() on all Append() calls
I20260806 06:30:19.057977 17482 tablet_bootstrap.cc:492] T ab6d3ecf31dd4645b797be5cf970a6ec P 07f80cff67044cb38d19f026639995e7: No bootstrap required, opened a new log
I20260806 06:30:19.058033 17482 ts_tablet_manager.cc:1397] T ab6d3ecf31dd4645b797be5cf970a6ec P 07f80cff67044cb38d19f026639995e7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260806 06:30:19.058158 17487 tablet_bootstrap.cc:492] T ab6d3ecf31dd4645b797be5cf970a6ec P c5ad55bc1785499d854288a27769fefe: No bootstrap required, opened a new log
I20260806 06:30:19.058218 17487 ts_tablet_manager.cc:1397] T ab6d3ecf31dd4645b797be5cf970a6ec P c5ad55bc1785499d854288a27769fefe: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260806 06:30:19.058791 17483 tablet_bootstrap.cc:492] T ab6d3ecf31dd4645b797be5cf970a6ec P 3b60d1e056b24cc49271bc406fac687f: No bootstrap required, opened a new log
I20260806 06:30:19.058866 17483 ts_tablet_manager.cc:1397] T ab6d3ecf31dd4645b797be5cf970a6ec P 3b60d1e056b24cc49271bc406fac687f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260806 06:30:19.060025 17482 raft_consensus.cc:348] T ab6d3ecf31dd4645b797be5cf970a6ec P 07f80cff67044cb38d19f026639995e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5ad55bc1785499d854288a27769fefe" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 33641 } } peers { permanent_uuid: "3b60d1e056b24cc49271bc406fac687f" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 42101 } } peers { permanent_uuid: "07f80cff67044cb38d19f026639995e7" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 36973 } }
I20260806 06:30:19.060163 17482 raft_consensus.cc:374] T ab6d3ecf31dd4645b797be5cf970a6ec P 07f80cff67044cb38d19f026639995e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 06:30:19.060194 17482 raft_consensus.cc:729] T ab6d3ecf31dd4645b797be5cf970a6ec P 07f80cff67044cb38d19f026639995e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07f80cff67044cb38d19f026639995e7, State: Initialized, Role: FOLLOWER
I20260806 06:30:19.060307 17482 consensus_queue.cc:260] T ab6d3ecf31dd4645b797be5cf970a6ec P 07f80cff67044cb38d19f026639995e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5ad55bc1785499d854288a27769fefe" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 33641 } } peers { permanent_uuid: "3b60d1e056b24cc49271bc406fac687f" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 42101 } } peers { permanent_uuid: "07f80cff67044cb38d19f026639995e7" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 36973 } }
I20260806 06:30:19.060550 17482 ts_tablet_manager.cc:1428] T ab6d3ecf31dd4645b797be5cf970a6ec P 07f80cff67044cb38d19f026639995e7: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260806 06:30:19.060592 17487 raft_consensus.cc:348] T ab6d3ecf31dd4645b797be5cf970a6ec P c5ad55bc1785499d854288a27769fefe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5ad55bc1785499d854288a27769fefe" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 33641 } } peers { permanent_uuid: "3b60d1e056b24cc49271bc406fac687f" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 42101 } } peers { permanent_uuid: "07f80cff67044cb38d19f026639995e7" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 36973 } }
I20260806 06:30:19.060745 17487 raft_consensus.cc:374] T ab6d3ecf31dd4645b797be5cf970a6ec P c5ad55bc1785499d854288a27769fefe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 06:30:19.060760 17461 heartbeater.cc:503] Master 127.21.12.190:35715 was elected leader, sending a full tablet report...
I20260806 06:30:19.060775 17487 raft_consensus.cc:729] T ab6d3ecf31dd4645b797be5cf970a6ec P c5ad55bc1785499d854288a27769fefe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5ad55bc1785499d854288a27769fefe, State: Initialized, Role: FOLLOWER
I20260806 06:30:19.060899 17487 consensus_queue.cc:260] T ab6d3ecf31dd4645b797be5cf970a6ec P c5ad55bc1785499d854288a27769fefe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5ad55bc1785499d854288a27769fefe" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 33641 } } peers { permanent_uuid: "3b60d1e056b24cc49271bc406fac687f" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 42101 } } peers { permanent_uuid: "07f80cff67044cb38d19f026639995e7" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 36973 } }
I20260806 06:30:19.061141 17487 ts_tablet_manager.cc:1428] T ab6d3ecf31dd4645b797be5cf970a6ec P c5ad55bc1785499d854288a27769fefe: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260806 06:30:19.061257 17081 heartbeater.cc:503] Master 127.21.12.190:35715 was elected leader, sending a full tablet report...
I20260806 06:30:19.061236 17483 raft_consensus.cc:348] T ab6d3ecf31dd4645b797be5cf970a6ec P 3b60d1e056b24cc49271bc406fac687f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5ad55bc1785499d854288a27769fefe" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 33641 } } peers { permanent_uuid: "3b60d1e056b24cc49271bc406fac687f" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 42101 } } peers { permanent_uuid: "07f80cff67044cb38d19f026639995e7" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 36973 } }
I20260806 06:30:19.061390 17483 raft_consensus.cc:374] T ab6d3ecf31dd4645b797be5cf970a6ec P 3b60d1e056b24cc49271bc406fac687f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 06:30:19.061432 17483 raft_consensus.cc:729] T ab6d3ecf31dd4645b797be5cf970a6ec P 3b60d1e056b24cc49271bc406fac687f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b60d1e056b24cc49271bc406fac687f, State: Initialized, Role: FOLLOWER
I20260806 06:30:19.061550 17483 consensus_queue.cc:260] T ab6d3ecf31dd4645b797be5cf970a6ec P 3b60d1e056b24cc49271bc406fac687f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5ad55bc1785499d854288a27769fefe" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 33641 } } peers { permanent_uuid: "3b60d1e056b24cc49271bc406fac687f" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 42101 } } peers { permanent_uuid: "07f80cff67044cb38d19f026639995e7" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 36973 } }
I20260806 06:30:19.061806 17483 ts_tablet_manager.cc:1428] T ab6d3ecf31dd4645b797be5cf970a6ec P 3b60d1e056b24cc49271bc406fac687f: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260806 06:30:19.061947 17262 heartbeater.cc:503] Master 127.21.12.190:35715 was elected leader, sending a full tablet report...
I20260806 06:30:19.119222 17492 raft_consensus.cc:482] T ab6d3ecf31dd4645b797be5cf970a6ec P 07f80cff67044cb38d19f026639995e7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 06:30:19.119347 17492 raft_consensus.cc:504] T ab6d3ecf31dd4645b797be5cf970a6ec P 07f80cff67044cb38d19f026639995e7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5ad55bc1785499d854288a27769fefe" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 33641 } } peers { permanent_uuid: "3b60d1e056b24cc49271bc406fac687f" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 42101 } } peers { permanent_uuid: "07f80cff67044cb38d19f026639995e7" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 36973 } }
I20260806 06:30:19.119652 17492 leader_election.cc:288] T ab6d3ecf31dd4645b797be5cf970a6ec P 07f80cff67044cb38d19f026639995e7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c5ad55bc1785499d854288a27769fefe (127.21.12.129:33641), 3b60d1e056b24cc49271bc406fac687f (127.21.12.130:42101)
I20260806 06:30:19.122615 17480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:19.119880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.131:51547 (local address 127.21.12.130:42101)
0806 06:30:19.119927 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0806 06:30:19.119929 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:30:19.120098 (+ 169us) server_negotiation.cc:407] Connection header received
0806 06:30:19.120169 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:30:19.120170 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:30:19.120180 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:30:19.120225 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 06:30:19.120362 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:19.120842 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:19.121880 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:19.122251 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:19.122363 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:30:19.122371 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0806 06:30:19.122502 (+ 131us) server_negotiation.cc:299] Negotiation successful
0806 06:30:19.122541 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260806 06:30:19.122612 17478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:19.119889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.131:40247 (local address 127.21.12.129:33641)
0806 06:30:19.119927 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0806 06:30:19.119929 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:30:19.119938 (+ 9us) server_negotiation.cc:407] Connection header received
0806 06:30:19.119969 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:30:19.119970 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:30:19.119982 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:30:19.120049 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 06:30:19.120205 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:19.120741 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:19.121804 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:19.122181 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:19.122234 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:30:19.122240 (+ 6us) server_negotiation.cc:1035] Waiting for connection context
0806 06:30:19.122494 (+ 254us) server_negotiation.cc:299] Negotiation successful
0806 06:30:19.122542 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260806 06:30:19.122623 17465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:19.119810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.129:33641 (local address 127.21.12.131:40247)
0806 06:30:19.119864 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0806 06:30:19.119875 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0806 06:30:19.119915 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:30:19.120053 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:30:19.120054 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:30:19.120055 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 06:30:19.120136 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:30:19.120138 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:30:19.120815 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:30:19.120816 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:30:19.121729 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:30:19.121731 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:30:19.122342 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:30:19.122343 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:30:19.122414 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:30:19.122418 (+ 4us) client_negotiation.cc:769] Sending connection context
0806 06:30:19.122475 (+ 57us) client_negotiation.cc:240] Negotiation successful
0806 06:30:19.122528 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36,"mutex_wait_us":12}
I20260806 06:30:19.122624 17496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:19.119826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.130:42101 (local address 127.21.12.131:51547)
0806 06:30:19.120007 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0806 06:30:19.120018 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0806 06:30:19.120104 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:30:19.120237 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:30:19.120238 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:30:19.120239 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 06:30:19.120302 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:30:19.120304 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:30:19.120915 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:30:19.120916 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:30:19.121805 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:30:19.121808 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:30:19.122373 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:30:19.122374 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:30:19.122442 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:30:19.122445 (+ 3us) client_negotiation.cc:769] Sending connection context
0806 06:30:19.122493 (+ 48us) client_negotiation.cc:240] Negotiation successful
0806 06:30:19.122521 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":113,"threads_started":1}
I20260806 06:30:19.122975 17018 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab6d3ecf31dd4645b797be5cf970a6ec" candidate_uuid: "07f80cff67044cb38d19f026639995e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5ad55bc1785499d854288a27769fefe" is_pre_election: true
I20260806 06:30:19.123137 17018 raft_consensus.cc:2454] T ab6d3ecf31dd4645b797be5cf970a6ec P c5ad55bc1785499d854288a27769fefe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 07f80cff67044cb38d19f026639995e7 in term 0.
I20260806 06:30:19.123178 17204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab6d3ecf31dd4645b797be5cf970a6ec" candidate_uuid: "07f80cff67044cb38d19f026639995e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b60d1e056b24cc49271bc406fac687f" is_pre_election: true
I20260806 06:30:19.123334 17204 raft_consensus.cc:2454] T ab6d3ecf31dd4645b797be5cf970a6ec P 3b60d1e056b24cc49271bc406fac687f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 07f80cff67044cb38d19f026639995e7 in term 0.
I20260806 06:30:19.123346 17294 leader_election.cc:302] T ab6d3ecf31dd4645b797be5cf970a6ec P 07f80cff67044cb38d19f026639995e7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07f80cff67044cb38d19f026639995e7, c5ad55bc1785499d854288a27769fefe; no voters:
I20260806 06:30:19.123514 17492 raft_consensus.cc:2781] T ab6d3ecf31dd4645b797be5cf970a6ec P 07f80cff67044cb38d19f026639995e7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 06:30:19.123572 17492 raft_consensus.cc:482] T ab6d3ecf31dd4645b797be5cf970a6ec P 07f80cff67044cb38d19f026639995e7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 06:30:19.123594 17492 raft_consensus.cc:3037] T ab6d3ecf31dd4645b797be5cf970a6ec P 07f80cff67044cb38d19f026639995e7 [term 0 FOLLOWER]: Advancing to term 1
I20260806 06:30:19.124269 17492 raft_consensus.cc:504] T ab6d3ecf31dd4645b797be5cf970a6ec P 07f80cff67044cb38d19f026639995e7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5ad55bc1785499d854288a27769fefe" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 33641 } } peers { permanent_uuid: "3b60d1e056b24cc49271bc406fac687f" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 42101 } } peers { permanent_uuid: "07f80cff67044cb38d19f026639995e7" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 36973 } }
I20260806 06:30:19.124454 17492 leader_election.cc:288] T ab6d3ecf31dd4645b797be5cf970a6ec P 07f80cff67044cb38d19f026639995e7 [CANDIDATE]: Term 1 election: Requested vote from peers c5ad55bc1785499d854288a27769fefe (127.21.12.129:33641), 3b60d1e056b24cc49271bc406fac687f (127.21.12.130:42101)
I20260806 06:30:19.124537 17204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab6d3ecf31dd4645b797be5cf970a6ec" candidate_uuid: "07f80cff67044cb38d19f026639995e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b60d1e056b24cc49271bc406fac687f"
I20260806 06:30:19.124601 17204 raft_consensus.cc:3037] T ab6d3ecf31dd4645b797be5cf970a6ec P 3b60d1e056b24cc49271bc406fac687f [term 0 FOLLOWER]: Advancing to term 1
I20260806 06:30:19.124616 17018 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab6d3ecf31dd4645b797be5cf970a6ec" candidate_uuid: "07f80cff67044cb38d19f026639995e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5ad55bc1785499d854288a27769fefe"
I20260806 06:30:19.124681 17018 raft_consensus.cc:3037] T ab6d3ecf31dd4645b797be5cf970a6ec P c5ad55bc1785499d854288a27769fefe [term 0 FOLLOWER]: Advancing to term 1
I20260806 06:30:19.125242 17204 raft_consensus.cc:2454] T ab6d3ecf31dd4645b797be5cf970a6ec P 3b60d1e056b24cc49271bc406fac687f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07f80cff67044cb38d19f026639995e7 in term 1.
I20260806 06:30:19.125401 17018 raft_consensus.cc:2454] T ab6d3ecf31dd4645b797be5cf970a6ec P c5ad55bc1785499d854288a27769fefe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07f80cff67044cb38d19f026639995e7 in term 1.
I20260806 06:30:19.125409 17293 leader_election.cc:302] T ab6d3ecf31dd4645b797be5cf970a6ec P 07f80cff67044cb38d19f026639995e7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07f80cff67044cb38d19f026639995e7, 3b60d1e056b24cc49271bc406fac687f; no voters:
I20260806 06:30:19.125573 17492 raft_consensus.cc:2781] T ab6d3ecf31dd4645b797be5cf970a6ec P 07f80cff67044cb38d19f026639995e7 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 06:30:19.125734 17492 raft_consensus.cc:686] T ab6d3ecf31dd4645b797be5cf970a6ec P 07f80cff67044cb38d19f026639995e7 [term 1 LEADER]: Becoming Leader. State: Replica: 07f80cff67044cb38d19f026639995e7, State: Running, Role: LEADER
I20260806 06:30:19.125823 17492 consensus_queue.cc:237] T ab6d3ecf31dd4645b797be5cf970a6ec P 07f80cff67044cb38d19f026639995e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5ad55bc1785499d854288a27769fefe" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 33641 } } peers { permanent_uuid: "3b60d1e056b24cc49271bc406fac687f" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 42101 } } peers { permanent_uuid: "07f80cff67044cb38d19f026639995e7" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 36973 } }
I20260806 06:30:19.126613 16806 catalog_manager.cc:5495] T ab6d3ecf31dd4645b797be5cf970a6ec P 07f80cff67044cb38d19f026639995e7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 07f80cff67044cb38d19f026639995e7 (127.21.12.131). New cstate: current_term: 1 leader_uuid: "07f80cff67044cb38d19f026639995e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5ad55bc1785499d854288a27769fefe" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 33641 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b60d1e056b24cc49271bc406fac687f" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 42101 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "07f80cff67044cb38d19f026639995e7" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 36973 } health_report { overall_health: HEALTHY } } }
I20260806 06:30:19.143805 17479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:19.141624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34546 (local address 127.21.12.131:36973)
0806 06:30:19.141668 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0806 06:30:19.141670 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:30:19.141679 (+ 9us) server_negotiation.cc:407] Connection header received
0806 06:30:19.141746 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:30:19.141747 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:30:19.141765 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:30:19.141821 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 06:30:19.141936 (+ 115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:19.142410 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:19.143097 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:19.143277 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:19.143330 (+ 53us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:30:19.143477 (+ 147us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 06:30:19.143563 (+ 86us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 06:30:19.143652 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0806 06:30:19.143711 (+ 59us) server_negotiation.cc:299] Negotiation successful
0806 06:30:19.143719 (+ 8us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260806 06:30:19.145429 17018 raft_consensus.cc:1261] T ab6d3ecf31dd4645b797be5cf970a6ec P c5ad55bc1785499d854288a27769fefe [term 1 FOLLOWER]: Refusing update from remote peer 07f80cff67044cb38d19f026639995e7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 06:30:19.145433 17204 raft_consensus.cc:1261] T ab6d3ecf31dd4645b797be5cf970a6ec P 3b60d1e056b24cc49271bc406fac687f [term 1 FOLLOWER]: Refusing update from remote peer 07f80cff67044cb38d19f026639995e7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 06:30:19.145717 17492 consensus_queue.cc:1035] T ab6d3ecf31dd4645b797be5cf970a6ec P 07f80cff67044cb38d19f026639995e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b60d1e056b24cc49271bc406fac687f" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 42101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 06:30:19.145849 17497 consensus_queue.cc:1035] T ab6d3ecf31dd4645b797be5cf970a6ec P 07f80cff67044cb38d19f026639995e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ad55bc1785499d854288a27769fefe" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 33641 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 06:30:19.148753 17502 mvcc.cc:205] Tried to move back new op lower bound from 7315447067217444864 to 7315447067140014080. Current Snapshot: MvccSnapshot[applied={T|T < 7315447067217444864}]
I20260806 06:30:19.151614 16806 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:59538:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 06:30:19.151710 16806 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:59538:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 06:30:19.152391 16806 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 996b5cec09f84daaba3aa73b9e82aade: Sending DeleteTablet for 3 replicas of tablet ab6d3ecf31dd4645b797be5cf970a6ec
I20260806 06:30:19.152786 17177 tablet_service.cc:1514] Processing DeleteTablet for tablet ab6d3ecf31dd4645b797be5cf970a6ec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 06:30:19 UTC) from {username='slave'} at 127.0.0.1:57346
I20260806 06:30:19.152968 21554 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHhLhT/build/release/bin/kudu with pid 16893
I20260806 06:30:19.153026 17356 tablet_service.cc:1514] Processing DeleteTablet for tablet ab6d3ecf31dd4645b797be5cf970a6ec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 06:30:19 UTC) from {username='slave'} at 127.0.0.1:34540
I20260806 06:30:19.161991 17513 tablet_replica.cc:330] T ab6d3ecf31dd4645b797be5cf970a6ec P 3b60d1e056b24cc49271bc406fac687f: stopping tablet replica
I20260806 06:30:19.162133 17513 raft_consensus.cc:2229] T ab6d3ecf31dd4645b797be5cf970a6ec P 3b60d1e056b24cc49271bc406fac687f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 06:30:19.162253 17513 raft_consensus.cc:2258] T ab6d3ecf31dd4645b797be5cf970a6ec P 3b60d1e056b24cc49271bc406fac687f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 06:30:19.163440 17513 ts_tablet_manager.cc:1905] T ab6d3ecf31dd4645b797be5cf970a6ec P 3b60d1e056b24cc49271bc406fac687f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 06:30:19.163893 17514 tablet_replica.cc:330] T ab6d3ecf31dd4645b797be5cf970a6ec P 07f80cff67044cb38d19f026639995e7: stopping tablet replica
I20260806 06:30:19.163993 17514 raft_consensus.cc:2229] T ab6d3ecf31dd4645b797be5cf970a6ec P 07f80cff67044cb38d19f026639995e7 [term 1 LEADER]: Raft consensus shutting down.
I20260806 06:30:19.164139 17514 raft_consensus.cc:2258] T ab6d3ecf31dd4645b797be5cf970a6ec P 07f80cff67044cb38d19f026639995e7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 06:30:19.165144 17513 ts_tablet_manager.cc:1918] T ab6d3ecf31dd4645b797be5cf970a6ec P 3b60d1e056b24cc49271bc406fac687f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 06:30:19.165225 17513 log.cc:1196] T ab6d3ecf31dd4645b797be5cf970a6ec P 3b60d1e056b24cc49271bc406fac687f: Deleting WAL directory at /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785997581049705-21554-0/minicluster-data/ts-1/wal/wals/ab6d3ecf31dd4645b797be5cf970a6ec
I20260806 06:30:19.165468 17513 ts_tablet_manager.cc:1939] T ab6d3ecf31dd4645b797be5cf970a6ec P 3b60d1e056b24cc49271bc406fac687f: Deleting consensus metadata
I20260806 06:30:19.166394 16783 catalog_manager.cc:4841] TS 3b60d1e056b24cc49271bc406fac687f (127.21.12.130:42101): tablet ab6d3ecf31dd4645b797be5cf970a6ec (table test-table [id=759c450ecadc4c5d82678cedf409ce47]) successfully deleted
I20260806 06:30:19.174652 17514 ts_tablet_manager.cc:1905] T ab6d3ecf31dd4645b797be5cf970a6ec P 07f80cff67044cb38d19f026639995e7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 06:30:19.176029 17514 ts_tablet_manager.cc:1918] T ab6d3ecf31dd4645b797be5cf970a6ec P 07f80cff67044cb38d19f026639995e7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 06:30:19.176100 17514 log.cc:1196] T ab6d3ecf31dd4645b797be5cf970a6ec P 07f80cff67044cb38d19f026639995e7: Deleting WAL directory at /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785997581049705-21554-0/minicluster-data/ts-2/wal/wals/ab6d3ecf31dd4645b797be5cf970a6ec
I20260806 06:30:19.176295 17514 ts_tablet_manager.cc:1939] T ab6d3ecf31dd4645b797be5cf970a6ec P 07f80cff67044cb38d19f026639995e7: Deleting consensus metadata
I20260806 06:30:19.176920 16786 catalog_manager.cc:4841] TS 07f80cff67044cb38d19f026639995e7 (127.21.12.131:36973): tablet ab6d3ecf31dd4645b797be5cf970a6ec (table test-table [id=759c450ecadc4c5d82678cedf409ce47]) successfully deleted
W20260806 06:30:19.180558 16784 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.12.129:33641 (error 108)
I20260806 06:30:19.180755 21554 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHhLhT/build/release/bin/kudu with pid 17084
I20260806 06:30:19.181367 17475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:19.180920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.129:33641 (local address 127.0.0.1:36902)
0806 06:30:19.181272 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0806 06:30:19.181329 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.129:33641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":324}
W20260806 06:30:19.182355 16784 catalog_manager.cc:4584] TS c5ad55bc1785499d854288a27769fefe (127.21.12.129:33641): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ab6d3ecf31dd4645b797be5cf970a6ec: Network error: Client connection negotiation failed: client connection to 127.21.12.129:33641: connect: Connection refused (error 111)
I20260806 06:30:19.188937 21554 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHhLhT/build/release/bin/kudu with pid 17268
I20260806 06:30:19.197628 21554 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHhLhT/build/release/bin/kudu with pid 16760
2026-08-06T06:30:19Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (716 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785997581049705-21554-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 06:30:19 dist-test-slave-4tdj krb5kdc[17520](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 06:30:19 dist-test-slave-4tdj krb5kdc[17520](info): set up 2 sockets
Aug 06 06:30:19 dist-test-slave-4tdj krb5kdc[17520](info): commencing operation
krb5kdc: starting...
W20260806 06:30:20.310468 21554 mini_kdc.cc:121] Time spent starting KDC: real 1.091s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785997581049705-21554-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785997581049705-21554-0/krb5kdc/test-user.keytab.
Aug 06 06:30:20 dist-test-slave-4tdj krb5kdc[17520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997820, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T06:30:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T06:30:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.12.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.12.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.12.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.190.keytab.
Entry for principal kudu/127.21.12.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.190.keytab.
I20260806 06:30:20.422242 21554 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHhLhT/build/release/bin/kudu
/tmp/dist-test-taskFHhLhT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785997581049705-21554-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785997581049705-21554-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785997581049705-21554-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785997581049705-21554-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.12.190:36137
--webserver_interface=127.21.12.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.190.keytab
--principal=kudu/127.21.12.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.148:35209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.12.190:36137
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785997581049705-21554-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785997581049705-21554-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785997581049705-21554-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 06:30:20.500722 17541 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 06:30:20.500923 17541 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 06:30:20.500952 17541 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 06:30:20.502068 17541 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 06:30:20.502107 17541 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 06:30:20.502125 17541 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 06:30:20.502141 17541 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 06:30:20.502156 17541 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 06:30:20.503618 17541 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.148:35209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785997581049705-21554-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785997581049705-21554-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.12.190:36137
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.190.keytab
--principal=kudu/127.21.12.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.12.190:36137
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785997581049705-21554-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.12.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17541
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785997581049705-21554-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 05:11:02 UTC on ccycloud.quasar-hvzbbz.root.comops.site
build id 4569617
I20260806 06:30:20.503861 17541 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 06:30:20.504055 17541 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 06:30:20.508616 17547 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:30:20.508625 17549 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:30:20.508628 17552 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:30:20.508874 17541 server_base.cc:1034] running on GCE node
I20260806 06:30:20.509097 17541 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 06:30:20.509352 17541 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 06:30:20.510494 17541 hybrid_clock.cc:648] HybridClock initialized: now 1785997820510481 us; error 28 us; skew 500 ppm
Aug 06 06:30:20 dist-test-slave-4tdj krb5kdc[17520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997820, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 06:30:20.512787 17541 init.cc:377] Logged in from keytab as kudu/127.21.12.190@KRBTEST.COM (short username kudu)
I20260806 06:30:20.513209 17541 webserver.cc:460] Webserver started at http://127.21.12.190:43593/ using document root <none> and password file <none>
I20260806 06:30:20.513415 17541 fs_manager.cc:359] Metadata directory not provided
I20260806 06:30:20.513463 17541 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 06:30:20.513587 17541 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 06:30:20.514606 17541 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785997581049705-21554-0/minicluster-data/master-0/data/instance:
uuid: "2f570376f25a452b883cb6a817b4ce26"
format_stamp: "Formatted at 2026-08-06 06:30:20 on dist-test-slave-4tdj"
server_key: "6987121cab9e6f81528e181b37ea6c96"
server_key_iv: "6a92fa2063f0c4fcb18a01061904d527"
server_key_version: "encryptionkey@0"
I20260806 06:30:20.514920 17541 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785997581049705-21554-0/minicluster-data/master-0/wal/instance:
uuid: "2f570376f25a452b883cb6a817b4ce26"
format_stamp: "Formatted at 2026-08-06 06:30:20 on dist-test-slave-4tdj"
server_key: "6987121cab9e6f81528e181b37ea6c96"
server_key_iv: "6a92fa2063f0c4fcb18a01061904d527"
server_key_version: "encryptionkey@0"
I20260806 06:30:20.516124 17541 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260806 06:30:20.516809 17565 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 06:30:20.516955 17541 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260806 06:30:20.517038 17541 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785997581049705-21554-0/minicluster-data/master-0/data,/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785997581049705-21554-0/minicluster-data/master-0/wal
uuid: "2f570376f25a452b883cb6a817b4ce26"
format_stamp: "Formatted at 2026-08-06 06:30:20 on dist-test-slave-4tdj"
server_key: "6987121cab9e6f81528e181b37ea6c96"
server_key_iv: "6a92fa2063f0c4fcb18a01061904d527"
server_key_version: "encryptionkey@0"
I20260806 06:30:20.517102 17541 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785997581049705-21554-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785997581049705-21554-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785997581049705-21554-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 06:30:20.544487 17541 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 06:30:20.546651 17541 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 06:30:20.546787 17541 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 06:30:20.550628 17541 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.190:36137
I20260806 06:30:20.550666 17653 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.190:36137 every 8 connection(s)
I20260806 06:30:20.550990 17541 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785997581049705-21554-0/minicluster-data/master-0/data/info.pb
I20260806 06:30:20.551862 17654 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 06:30:20.554178 17654 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f570376f25a452b883cb6a817b4ce26: Bootstrap starting.
I20260806 06:30:20.554775 17654 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2f570376f25a452b883cb6a817b4ce26: Neither blocks nor log segments found. Creating new log.
I20260806 06:30:20.555043 17654 log.cc:824] T 00000000000000000000000000000000 P 2f570376f25a452b883cb6a817b4ce26: Log is configured to *not* fsync() on all Append() calls
I20260806 06:30:20.555745 17654 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f570376f25a452b883cb6a817b4ce26: No bootstrap required, opened a new log
I20260806 06:30:20.556455 21554 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHhLhT/build/release/bin/kudu as pid 17541
I20260806 06:30:20.556568 21554 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785997581049705-21554-0/minicluster-data/master-0/wal/instance
I20260806 06:30:20.556744 21554 external_mini_cluster.cc:1428] Setting key 43ad383681b445ab78a432311dc046bc
I20260806 06:30:20.557546 17654 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2f570376f25a452b883cb6a817b4ce26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f570376f25a452b883cb6a817b4ce26" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 36137 } }
I20260806 06:30:20.557685 17654 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2f570376f25a452b883cb6a817b4ce26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 06:30:20.557716 17654 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2f570376f25a452b883cb6a817b4ce26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f570376f25a452b883cb6a817b4ce26, State: Initialized, Role: FOLLOWER
I20260806 06:30:20.557845 17654 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2f570376f25a452b883cb6a817b4ce26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f570376f25a452b883cb6a817b4ce26" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 36137 } }
I20260806 06:30:20.557932 17654 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2f570376f25a452b883cb6a817b4ce26 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 06:30:20.557969 17654 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2f570376f25a452b883cb6a817b4ce26 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 06:30:20.558019 17654 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2f570376f25a452b883cb6a817b4ce26 [term 0 FOLLOWER]: Advancing to term 1
I20260806 06:30:20.559012 17654 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2f570376f25a452b883cb6a817b4ce26 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f570376f25a452b883cb6a817b4ce26" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 36137 } }
I20260806 06:30:20.559103 17654 leader_election.cc:302] T 00000000000000000000000000000000 P 2f570376f25a452b883cb6a817b4ce26 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2f570376f25a452b883cb6a817b4ce26; no voters:
I20260806 06:30:20.559247 17654 leader_election.cc:288] T 00000000000000000000000000000000 P 2f570376f25a452b883cb6a817b4ce26 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 06:30:20.559288 17662 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2f570376f25a452b883cb6a817b4ce26 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 06:30:20.559428 17662 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2f570376f25a452b883cb6a817b4ce26 [term 1 LEADER]: Becoming Leader. State: Replica: 2f570376f25a452b883cb6a817b4ce26, State: Running, Role: LEADER
I20260806 06:30:20.559485 17654 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2f570376f25a452b883cb6a817b4ce26 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 06:30:20.559509 17662 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2f570376f25a452b883cb6a817b4ce26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f570376f25a452b883cb6a817b4ce26" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 36137 } }
I20260806 06:30:20.559821 17662 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f570376f25a452b883cb6a817b4ce26 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2f570376f25a452b883cb6a817b4ce26. Latest consensus state: current_term: 1 leader_uuid: "2f570376f25a452b883cb6a817b4ce26" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f570376f25a452b883cb6a817b4ce26" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 36137 } } }
I20260806 06:30:20.559899 17662 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f570376f25a452b883cb6a817b4ce26 [sys.catalog]: This master's current role is: LEADER
I20260806 06:30:20.559828 17665 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f570376f25a452b883cb6a817b4ce26 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2f570376f25a452b883cb6a817b4ce26" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f570376f25a452b883cb6a817b4ce26" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 36137 } } }
I20260806 06:30:20.560122 17665 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f570376f25a452b883cb6a817b4ce26 [sys.catalog]: This master's current role is: LEADER
I20260806 06:30:20.560837 17667 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 06:30:20.561519 17667 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 06:30:20.563087 17667 catalog_manager.cc:1347] Generated new cluster ID: c2e05e36520740a9bc79237ba57723a2
I20260806 06:30:20.563135 17667 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 06 06:30:20 dist-test-slave-4tdj krb5kdc[17520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997820, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.12.190@KRBTEST.COM
I20260806 06:30:20.568071 17659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:20.557235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33042 (local address 127.21.12.190:36137)
0806 06:30:20.557415 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0806 06:30:20.557422 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 06:30:20.557447 (+ 25us) server_negotiation.cc:407] Connection header received
0806 06:30:20.557570 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:30:20.557575 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:30:20.557618 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:30:20.557740 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:30:20.558596 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:20.559675 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:20.560709 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:20.561282 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:20.561360 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:30:20.564040 (+ 2680us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:30:20.564044 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:30:20.564044 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 06:30:20.564056 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:30:20.566001 (+ 1945us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:30:20.566420 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:30:20.566423 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:30:20.566424 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:30:20.566563 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:30:20.566782 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:30:20.566783 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:30:20.566784 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:30:20.567103 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:30:20.567245 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0806 06:30:20.567736 (+ 491us) server_negotiation.cc:299] Negotiation successful
0806 06:30:20.567875 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":80,"threads_started":1}
I20260806 06:30:20.569449 17667 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 06:30:20.569940 17667 catalog_manager.cc:1504] Loading token signing keys...
I20260806 06:30:20.572142 17667 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2f570376f25a452b883cb6a817b4ce26: Generated new TSK 0
I20260806 06:30:20.572286 17667 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.21.12.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.12.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.12.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.129.keytab.
Entry for principal kudu/127.21.12.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.129.keytab.
I20260806 06:30:20.600725 21554 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHhLhT/build/release/bin/kudu
/tmp/dist-test-taskFHhLhT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785997581049705-21554-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785997581049705-21554-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785997581049705-21554-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785997581049705-21554-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.12.129:0
--local_ip_for_outbound_sockets=127.21.12.129
--webserver_interface=127.21.12.129
--webserver_port=0
--tserver_master_addrs=127.21.12.190:36137
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.129.keytab
--principal=kudu/127.21.12.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.148:35209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785997581049705-21554-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785997581049705-21554-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785997581049705-21554-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 06:30:20.681360 17689 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 06:30:20.681571 17689 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 06:30:20.681599 17689 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 06:30:20.682791 17689 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 06:30:20.682832 17689 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 06:30:20.682857 17689 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.12.129
I20260806 06:30:20.684335 17689 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.148:35209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785997581049705-21554-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785997581049705-21554-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.129.keytab
--principal=kudu/127.21.12.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.12.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785997581049705-21554-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.12.129
--webserver_port=0
--tserver_master_addrs=127.21.12.190:36137
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17689
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.12.129
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785997581049705-21554-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 05:11:02 UTC on ccycloud.quasar-hvzbbz.root.comops.site
build id 4569617
I20260806 06:30:20.684569 17689 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 06:30:20.684790 17689 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 06:30:20.689863 17696 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:30:20.689860 17698 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:30:20.689874 17702 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:30:20.689886 17689 server_base.cc:1034] running on GCE node
I20260806 06:30:20.690241 17689 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 06:30:20.690496 17689 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 06:30:20.691637 17689 hybrid_clock.cc:648] HybridClock initialized: now 1785997820691627 us; error 33 us; skew 500 ppm
Aug 06 06:30:20 dist-test-slave-4tdj krb5kdc[17520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997820, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 06:30:20.694623 17689 init.cc:377] Logged in from keytab as kudu/127.21.12.129@KRBTEST.COM (short username kudu)
I20260806 06:30:20.695087 17689 webserver.cc:460] Webserver started at http://127.21.12.129:46241/ using document root <none> and password file <none>
I20260806 06:30:20.695322 17689 fs_manager.cc:359] Metadata directory not provided
I20260806 06:30:20.695369 17689 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 06:30:20.695508 17689 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 06:30:20.696574 17689 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785997581049705-21554-0/minicluster-data/ts-0/data/instance:
uuid: "bbcbea95d2204210bed7a36961ec71af"
format_stamp: "Formatted at 2026-08-06 06:30:20 on dist-test-slave-4tdj"
server_key: "ddc5cce48565558f38ad19d96e11f268"
server_key_iv: "71f543da376ee672824b1e63f59927fa"
server_key_version: "encryptionkey@0"
I20260806 06:30:20.696913 17689 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785997581049705-21554-0/minicluster-data/ts-0/wal/instance:
uuid: "bbcbea95d2204210bed7a36961ec71af"
format_stamp: "Formatted at 2026-08-06 06:30:20 on dist-test-slave-4tdj"
server_key: "ddc5cce48565558f38ad19d96e11f268"
server_key_iv: "71f543da376ee672824b1e63f59927fa"
server_key_version: "encryptionkey@0"
I20260806 06:30:20.698210 17689 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260806 06:30:20.698958 17709 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 06:30:20.699103 17689 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260806 06:30:20.699194 17689 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785997581049705-21554-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785997581049705-21554-0/minicluster-data/ts-0/wal
uuid: "bbcbea95d2204210bed7a36961ec71af"
format_stamp: "Formatted at 2026-08-06 06:30:20 on dist-test-slave-4tdj"
server_key: "ddc5cce48565558f38ad19d96e11f268"
server_key_iv: "71f543da376ee672824b1e63f59927fa"
server_key_version: "encryptionkey@0"
I20260806 06:30:20.699258 17689 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785997581049705-21554-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785997581049705-21554-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785997581049705-21554-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 06:30:20.710906 17689 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 06:30:20.712733 17689 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 06:30:20.712857 17689 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 06:30:20.713053 17689 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 06:30:20.713337 17689 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 06:30:20.713379 17689 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 06:30:20.713406 17689 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 06:30:20.713429 17689 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 06:30:20.719141 17689 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.129:40075
I20260806 06:30:20.719202 17876 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.129:40075 every 8 connection(s)
I20260806 06:30:20.719551 17689 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785997581049705-21554-0/minicluster-data/ts-0/data/info.pb
Aug 06 06:30:20 dist-test-slave-4tdj krb5kdc[17520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997820, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.129@KRBTEST.COM for kudu/127.21.12.190@KRBTEST.COM
I20260806 06:30:20.725663 21554 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHhLhT/build/release/bin/kudu as pid 17689
I20260806 06:30:20.725785 21554 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785997581049705-21554-0/minicluster-data/ts-0/wal/instance
I20260806 06:30:20.725975 21554 external_mini_cluster.cc:1428] Setting key f7efe6ceaf4f7fa5128733f3443bd842
I20260806 06:30:20.728152 17659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:20.720242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.129:51275 (local address 127.21.12.190:36137)
0806 06:30:20.720318 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0806 06:30:20.720320 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:30:20.720479 (+ 159us) server_negotiation.cc:407] Connection header received
0806 06:30:20.720925 (+ 446us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:30:20.720926 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:30:20.720940 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:30:20.720988 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:30:20.721861 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:20.722411 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:20.723344 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:20.723541 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:20.723584 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:30:20.725506 (+ 1922us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:30:20.725507 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:30:20.725507 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 06:30:20.725515 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:30:20.727008 (+ 1493us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:30:20.727345 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:30:20.727346 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:30:20.727346 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:30:20.727406 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:30:20.727594 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:30:20.727594 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:30:20.727594 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:30:20.727721 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:30:20.727776 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0806 06:30:20.727983 (+ 207us) server_negotiation.cc:299] Negotiation successful
0806 06:30:20.728064 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260806 06:30:20.728237 17879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:20.720217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:36137 (local address 127.21.12.129:51275)
0806 06:30:20.720367 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0806 06:30:20.720402 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0806 06:30:20.720851 (+ 449us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:30:20.721046 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:30:20.721050 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:30:20.721603 (+ 553us) client_negotiation.cc:189] Negotiated authn=SASL
0806 06:30:20.721784 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:30:20.721789 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:30:20.722491 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:30:20.722492 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:30:20.723282 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:30:20.723285 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:30:20.723592 (+ 307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:30:20.723592 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:30:20.723685 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:30:20.724118 (+ 433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 06:30:20.724127 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0806 06:30:20.725418 (+ 1291us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 06:30:20.727076 (+ 1658us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:30:20.727083 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:30:20.727084 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:30:20.727282 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:30:20.727459 (+ 177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:30:20.727460 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:30:20.727460 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:30:20.727547 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:30:20.727775 (+ 228us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 06:30:20.727778 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 06:30:20.727880 (+ 102us) client_negotiation.cc:769] Sending connection context
0806 06:30:20.727953 (+ 73us) client_negotiation.cc:240] Negotiation successful
0806 06:30:20.728072 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":73,"threads_started":1}
I20260806 06:30:20.728725 17877 heartbeater.cc:347] Connected to a master server at 127.21.12.190:36137
I20260806 06:30:20.728821 17877 heartbeater.cc:464] Registering TS with master...
I20260806 06:30:20.729020 17877 heartbeater.cc:511] Master 127.21.12.190:36137 requested a full tablet report, sending...
I20260806 06:30:20.729535 17589 ts_manager.cc:194] Registered new tserver with Master: bbcbea95d2204210bed7a36961ec71af (127.21.12.129:40075)
I20260806 06:30:20.730216 17589 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.12.129@KRBTEST.COM'} at 127.21.12.129:51275
WARNING: no policy specified for kudu/127.21.12.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.12.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.12.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.130.keytab.
Entry for principal kudu/127.21.12.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.130.keytab.
I20260806 06:30:20.750438 21554 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHhLhT/build/release/bin/kudu
/tmp/dist-test-taskFHhLhT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785997581049705-21554-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785997581049705-21554-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785997581049705-21554-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785997581049705-21554-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.12.130:0
--local_ip_for_outbound_sockets=127.21.12.130
--webserver_interface=127.21.12.130
--webserver_port=0
--tserver_master_addrs=127.21.12.190:36137
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.130.keytab
--principal=kudu/127.21.12.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.148:35209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785997581049705-21554-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785997581049705-21554-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785997581049705-21554-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 06:30:20.830010 17883 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 06:30:20.830192 17883 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 06:30:20.830214 17883 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 06:30:20.831331 17883 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 06:30:20.831370 17883 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 06:30:20.831395 17883 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.12.130
I20260806 06:30:20.832782 17883 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.148:35209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785997581049705-21554-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785997581049705-21554-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.130.keytab
--principal=kudu/127.21.12.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.12.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785997581049705-21554-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.12.130
--webserver_port=0
--tserver_master_addrs=127.21.12.190:36137
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17883
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.12.130
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785997581049705-21554-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 05:11:02 UTC on ccycloud.quasar-hvzbbz.root.comops.site
build id 4569617
I20260806 06:30:20.832981 17883 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 06:30:20.833187 17883 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260806 06:30:20.837911 17883 server_base.cc:1034] running on GCE node
W20260806 06:30:20.837857 17893 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:30:20.837864 17890 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:30:20.837864 17891 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:30:20.838232 17883 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 06:30:20.838466 17883 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 06:30:20.839622 17883 hybrid_clock.cc:648] HybridClock initialized: now 1785997820839608 us; error 28 us; skew 500 ppm
Aug 06 06:30:20 dist-test-slave-4tdj krb5kdc[17520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997820, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 06:30:20.842031 17883 init.cc:377] Logged in from keytab as kudu/127.21.12.130@KRBTEST.COM (short username kudu)
I20260806 06:30:20.842489 17883 webserver.cc:460] Webserver started at http://127.21.12.130:43045/ using document root <none> and password file <none>
I20260806 06:30:20.842664 17883 fs_manager.cc:359] Metadata directory not provided
I20260806 06:30:20.842702 17883 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 06:30:20.842796 17883 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 06:30:20.843735 17883 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785997581049705-21554-0/minicluster-data/ts-1/data/instance:
uuid: "ae7a64cd5c614e969183aca131d2962a"
format_stamp: "Formatted at 2026-08-06 06:30:20 on dist-test-slave-4tdj"
server_key: "28c21ff28fc136707dc5d1145ea5a3d6"
server_key_iv: "d7eee272aac00163f09f89802f0705d0"
server_key_version: "encryptionkey@0"
I20260806 06:30:20.844049 17883 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785997581049705-21554-0/minicluster-data/ts-1/wal/instance:
uuid: "ae7a64cd5c614e969183aca131d2962a"
format_stamp: "Formatted at 2026-08-06 06:30:20 on dist-test-slave-4tdj"
server_key: "28c21ff28fc136707dc5d1145ea5a3d6"
server_key_iv: "d7eee272aac00163f09f89802f0705d0"
server_key_version: "encryptionkey@0"
I20260806 06:30:20.845325 17883 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260806 06:30:20.845968 17903 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 06:30:20.846117 17883 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260806 06:30:20.846197 17883 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785997581049705-21554-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785997581049705-21554-0/minicluster-data/ts-1/wal
uuid: "ae7a64cd5c614e969183aca131d2962a"
format_stamp: "Formatted at 2026-08-06 06:30:20 on dist-test-slave-4tdj"
server_key: "28c21ff28fc136707dc5d1145ea5a3d6"
server_key_iv: "d7eee272aac00163f09f89802f0705d0"
server_key_version: "encryptionkey@0"
I20260806 06:30:20.846262 17883 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785997581049705-21554-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785997581049705-21554-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785997581049705-21554-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 06:30:20.874498 17883 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 06:30:20.876236 17883 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 06:30:20.876361 17883 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 06:30:20.876595 17883 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 06:30:20.876890 17883 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 06:30:20.876924 17883 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 06:30:20.876946 17883 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 06:30:20.876968 17883 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 06:30:20.882308 17883 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.130:38183
I20260806 06:30:20.882359 18070 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.130:38183 every 8 connection(s)
I20260806 06:30:20.882647 17883 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785997581049705-21554-0/minicluster-data/ts-1/data/info.pb
I20260806 06:30:20.884979 21554 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHhLhT/build/release/bin/kudu as pid 17883
I20260806 06:30:20.885108 21554 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785997581049705-21554-0/minicluster-data/ts-1/wal/instance
I20260806 06:30:20.885300 21554 external_mini_cluster.cc:1428] Setting key 02e835d8a5eb1c5a57effb3e748f89fc
Aug 06 06:30:20 dist-test-slave-4tdj krb5kdc[17520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997820, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.130@KRBTEST.COM for kudu/127.21.12.190@KRBTEST.COM
I20260806 06:30:20.892171 17659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:20.883275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.130:56573 (local address 127.21.12.190:36137)
0806 06:30:20.883332 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0806 06:30:20.883334 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:30:20.883525 (+ 191us) server_negotiation.cc:407] Connection header received
0806 06:30:20.883990 (+ 465us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:30:20.883991 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:30:20.884002 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:30:20.884051 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:30:20.885011 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:20.885680 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:20.887018 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:20.887208 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:20.887262 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:30:20.889471 (+ 2209us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:30:20.889472 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:30:20.889472 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 06:30:20.889480 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:30:20.890948 (+ 1468us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:30:20.891308 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:30:20.891310 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:30:20.891310 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:30:20.891370 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:30:20.891578 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:30:20.891578 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:30:20.891579 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:30:20.891718 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:30:20.891790 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0806 06:30:20.891993 (+ 203us) server_negotiation.cc:299] Negotiation successful
0806 06:30:20.892075 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260806 06:30:20.892244 18075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:20.883267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:36137 (local address 127.21.12.130:56573)
0806 06:30:20.883416 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0806 06:30:20.883450 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0806 06:30:20.883913 (+ 463us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:30:20.884120 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:30:20.884123 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:30:20.884678 (+ 555us) client_negotiation.cc:189] Negotiated authn=SASL
0806 06:30:20.884929 (+ 251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:30:20.884936 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:30:20.885769 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:30:20.885770 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:30:20.886936 (+ 1166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:30:20.886939 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:30:20.887286 (+ 347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:30:20.887287 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:30:20.887417 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:30:20.887919 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 06:30:20.887927 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0806 06:30:20.889379 (+ 1452us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 06:30:20.891022 (+ 1643us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:30:20.891029 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:30:20.891031 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:30:20.891241 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:30:20.891433 (+ 192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:30:20.891433 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:30:20.891433 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:30:20.891523 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:30:20.891777 (+ 254us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 06:30:20.891780 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 06:30:20.891881 (+ 101us) client_negotiation.cc:769] Sending connection context
0806 06:30:20.891954 (+ 73us) client_negotiation.cc:240] Negotiation successful
0806 06:30:20.892067 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":75,"threads_started":1}
I20260806 06:30:20.892683 18071 heartbeater.cc:347] Connected to a master server at 127.21.12.190:36137
I20260806 06:30:20.892774 18071 heartbeater.cc:464] Registering TS with master...
I20260806 06:30:20.892938 18071 heartbeater.cc:511] Master 127.21.12.190:36137 requested a full tablet report, sending...
I20260806 06:30:20.893296 17589 ts_manager.cc:194] Registered new tserver with Master: ae7a64cd5c614e969183aca131d2962a (127.21.12.130:38183)
I20260806 06:30:20.893805 17589 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.12.130@KRBTEST.COM'} at 127.21.12.130:56573
WARNING: no policy specified for kudu/127.21.12.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.12.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.12.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.131.keytab.
Entry for principal kudu/127.21.12.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.131.keytab.
I20260806 06:30:20.910938 21554 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHhLhT/build/release/bin/kudu
/tmp/dist-test-taskFHhLhT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785997581049705-21554-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785997581049705-21554-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785997581049705-21554-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785997581049705-21554-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.12.131:0
--local_ip_for_outbound_sockets=127.21.12.131
--webserver_interface=127.21.12.131
--webserver_port=0
--tserver_master_addrs=127.21.12.190:36137
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.131.keytab
--principal=kudu/127.21.12.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.148:35209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785997581049705-21554-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785997581049705-21554-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785997581049705-21554-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 06:30:20.991173 18080 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 06:30:20.991367 18080 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 06:30:20.991397 18080 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 06:30:20.992514 18080 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 06:30:20.992553 18080 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 06:30:20.992589 18080 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.12.131
I20260806 06:30:20.994009 18080 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.148:35209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785997581049705-21554-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785997581049705-21554-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.131.keytab
--principal=kudu/127.21.12.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.12.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785997581049705-21554-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.12.131
--webserver_port=0
--tserver_master_addrs=127.21.12.190:36137
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18080
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.12.131
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785997581049705-21554-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 05:11:02 UTC on ccycloud.quasar-hvzbbz.root.comops.site
build id 4569617
I20260806 06:30:20.994248 18080 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 06:30:20.994467 18080 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 06:30:20.999215 18091 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:30:20.999241 18087 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:30:20.999243 18088 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:30:20.999518 18080 server_base.cc:1034] running on GCE node
I20260806 06:30:20.999718 18080 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 06:30:20.999917 18080 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 06:30:21.001065 18080 hybrid_clock.cc:648] HybridClock initialized: now 1785997821001043 us; error 36 us; skew 500 ppm
Aug 06 06:30:20 dist-test-slave-4tdj krb5kdc[17520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997820, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 06:30:21.003597 18080 init.cc:377] Logged in from keytab as kudu/127.21.12.131@KRBTEST.COM (short username kudu)
I20260806 06:30:21.003989 18080 webserver.cc:460] Webserver started at http://127.21.12.131:36045/ using document root <none> and password file <none>
I20260806 06:30:21.004195 18080 fs_manager.cc:359] Metadata directory not provided
I20260806 06:30:21.004243 18080 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 06:30:21.004369 18080 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 06:30:21.005414 18080 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785997581049705-21554-0/minicluster-data/ts-2/data/instance:
uuid: "1d4c54a433a34e7d91d2176fac140297"
format_stamp: "Formatted at 2026-08-06 06:30:20 on dist-test-slave-4tdj"
server_key: "89408062e56571d68a8de7bad8f4a6f3"
server_key_iv: "e9f5e43e663a37ed932c0221d5e10e39"
server_key_version: "encryptionkey@0"
I20260806 06:30:21.005738 18080 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785997581049705-21554-0/minicluster-data/ts-2/wal/instance:
uuid: "1d4c54a433a34e7d91d2176fac140297"
format_stamp: "Formatted at 2026-08-06 06:30:20 on dist-test-slave-4tdj"
server_key: "89408062e56571d68a8de7bad8f4a6f3"
server_key_iv: "e9f5e43e663a37ed932c0221d5e10e39"
server_key_version: "encryptionkey@0"
I20260806 06:30:21.007032 18080 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260806 06:30:21.007695 18100 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 06:30:21.007851 18080 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260806 06:30:21.007937 18080 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785997581049705-21554-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785997581049705-21554-0/minicluster-data/ts-2/wal
uuid: "1d4c54a433a34e7d91d2176fac140297"
format_stamp: "Formatted at 2026-08-06 06:30:20 on dist-test-slave-4tdj"
server_key: "89408062e56571d68a8de7bad8f4a6f3"
server_key_iv: "e9f5e43e663a37ed932c0221d5e10e39"
server_key_version: "encryptionkey@0"
I20260806 06:30:21.007993 18080 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785997581049705-21554-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785997581049705-21554-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785997581049705-21554-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 06:30:21.017125 18080 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 06:30:21.018780 18080 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 06:30:21.018903 18080 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 06:30:21.019102 18080 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 06:30:21.019403 18080 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 06:30:21.019445 18080 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 06:30:21.019472 18080 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 06:30:21.019502 18080 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 06:30:21.026432 18080 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.131:45381
I20260806 06:30:21.026484 18265 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.131:45381 every 8 connection(s)
I20260806 06:30:21.026773 18080 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785997581049705-21554-0/minicluster-data/ts-2/data/info.pb
Aug 06 06:30:21 dist-test-slave-4tdj krb5kdc[17520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997820, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.131@KRBTEST.COM for kudu/127.21.12.190@KRBTEST.COM
I20260806 06:30:21.035090 17659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:21.027437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.131:45523 (local address 127.21.12.190:36137)
0806 06:30:21.027496 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0806 06:30:21.027498 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:30:21.027678 (+ 180us) server_negotiation.cc:407] Connection header received
0806 06:30:21.028137 (+ 459us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:30:21.028138 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:30:21.028148 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:30:21.028198 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:30:21.029033 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:21.029515 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:21.030514 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:21.030712 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:21.030770 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:30:21.032711 (+ 1941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:30:21.032713 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:30:21.032713 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 06:30:21.032720 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:30:21.033832 (+ 1112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:30:21.034181 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:30:21.034182 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:30:21.034183 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:30:21.034242 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:30:21.034476 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:30:21.034476 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:30:21.034477 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:30:21.034588 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:30:21.034645 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0806 06:30:21.034891 (+ 246us) server_negotiation.cc:299] Negotiation successful
0806 06:30:21.034982 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260806 06:30:21.035207 18269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:21.027416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:36137 (local address 127.21.12.131:45523)
0806 06:30:21.027563 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0806 06:30:21.027602 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0806 06:30:21.028061 (+ 459us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:30:21.028250 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:30:21.028254 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:30:21.028784 (+ 530us) client_negotiation.cc:189] Negotiated authn=SASL
0806 06:30:21.028963 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:30:21.028968 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:30:21.029594 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:30:21.029596 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:30:21.030417 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:30:21.030420 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:30:21.030784 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:30:21.030785 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:30:21.030916 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:30:21.031304 (+ 388us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 06:30:21.031311 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0806 06:30:21.032622 (+ 1311us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 06:30:21.033905 (+ 1283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:30:21.033912 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:30:21.033914 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:30:21.034114 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:30:21.034314 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:30:21.034314 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:30:21.034315 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:30:21.034419 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:30:21.034642 (+ 223us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 06:30:21.034648 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 06:30:21.034775 (+ 127us) client_negotiation.cc:769] Sending connection context
0806 06:30:21.034859 (+ 84us) client_negotiation.cc:240] Negotiation successful
0806 06:30:21.035005 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":80,"threads_started":1}
I20260806 06:30:21.035629 21554 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHhLhT/build/release/bin/kudu as pid 18080
I20260806 06:30:21.035650 18266 heartbeater.cc:347] Connected to a master server at 127.21.12.190:36137
I20260806 06:30:21.035740 18266 heartbeater.cc:464] Registering TS with master...
I20260806 06:30:21.035743 21554 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785997581049705-21554-0/minicluster-data/ts-2/wal/instance
I20260806 06:30:21.035921 18266 heartbeater.cc:511] Master 127.21.12.190:36137 requested a full tablet report, sending...
I20260806 06:30:21.035938 21554 external_mini_cluster.cc:1428] Setting key a36aaa48cf4f5bfca0a7cd90f2de8cd9
I20260806 06:30:21.036332 17588 ts_manager.cc:194] Registered new tserver with Master: 1d4c54a433a34e7d91d2176fac140297 (127.21.12.131:45381)
I20260806 06:30:21.036772 17588 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.21.12.131@KRBTEST.COM'} at 127.21.12.131:45523
I20260806 06:30:21.036774 21554 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 06:30:21.043159 17659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:21.038159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33048 (local address 127.21.12.190:36137)
0806 06:30:21.038224 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0806 06:30:21.038225 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0806 06:30:21.038341 (+ 116us) server_negotiation.cc:407] Connection header received
0806 06:30:21.038399 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:30:21.038400 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:30:21.038408 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:30:21.038446 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:30:21.038933 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:21.039370 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:21.040017 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:21.040194 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:21.040244 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:30:21.041240 (+ 996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:30:21.041241 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:30:21.041242 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 06:30:21.041248 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:30:21.042261 (+ 1013us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:30:21.042596 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:30:21.042597 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:30:21.042597 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:30:21.042646 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:30:21.042787 (+ 141us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:30:21.042787 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:30:21.042787 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:30:21.042907 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:30:21.042975 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0806 06:30:21.043044 (+ 69us) server_negotiation.cc:299] Negotiation successful
0806 06:30:21.043101 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260806 06:30:21.044694 17588 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33048:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 06:30:21.046201 17588 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 06:30:21.051950 18278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:21.048137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.131:45381 (local address 127.0.0.1:35726)
0806 06:30:21.048314 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0806 06:30:21.048354 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0806 06:30:21.048415 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:30:21.048639 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:30:21.048643 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:30:21.048645 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 06:30:21.048710 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:30:21.048712 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:30:21.049787 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:30:21.049788 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:30:21.050831 (+ 1043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:30:21.050834 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:30:21.051678 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:30:21.051679 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:30:21.051753 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:30:21.051766 (+ 13us) client_negotiation.cc:769] Sending connection context
0806 06:30:21.051823 (+ 57us) client_negotiation.cc:240] Negotiation successful
0806 06:30:21.051874 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"mutex_wait_us":1,"thread_start_us":75,"threads_started":1}
I20260806 06:30:21.051963 18280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:21.048287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35726 (local address 127.21.12.131:45381)
0806 06:30:21.048453 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0806 06:30:21.048457 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 06:30:21.048467 (+ 10us) server_negotiation.cc:407] Connection header received
0806 06:30:21.048502 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:30:21.048504 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:30:21.048520 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:30:21.048629 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 06:30:21.048788 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:21.049658 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:21.050924 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:21.051612 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:21.051668 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:30:21.051727 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0806 06:30:21.051847 (+ 120us) server_negotiation.cc:299] Negotiation successful
0806 06:30:21.051885 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":76,"threads_started":1}
I20260806 06:30:21.052471 18279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:21.048140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.130:38183 (local address 127.0.0.1:60836)
0806 06:30:21.048649 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0806 06:30:21.048659 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0806 06:30:21.048709 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:30:21.049017 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:30:21.049018 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:30:21.049020 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 06:30:21.049061 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:30:21.049063 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:30:21.050380 (+ 1317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:30:21.050381 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:30:21.051238 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:30:21.051241 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:30:21.052237 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:30:21.052238 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:30:21.052297 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:30:21.052301 (+ 4us) client_negotiation.cc:769] Sending connection context
0806 06:30:21.052353 (+ 52us) client_negotiation.cc:240] Negotiation successful
0806 06:30:21.052394 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":495,"mutex_wait_us":5,"thread_start_us":71,"threads_started":1}
I20260806 06:30:21.052492 18282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:21.048341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60836 (local address 127.21.12.130:38183)
0806 06:30:21.048557 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0806 06:30:21.048561 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 06:30:21.048711 (+ 150us) server_negotiation.cc:407] Connection header received
0806 06:30:21.048781 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:30:21.048783 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:30:21.048799 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:30:21.048943 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 06:30:21.049156 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:21.050272 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:21.051308 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:21.051940 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:21.052141 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:30:21.052200 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0806 06:30:21.052370 (+ 170us) server_negotiation.cc:299] Negotiation successful
0806 06:30:21.052415 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":184,"thread_start_us":99,"threads_started":1}
I20260806 06:30:21.052635 18284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:21.048417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55726 (local address 127.21.12.129:40075)
0806 06:30:21.048579 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0806 06:30:21.048583 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 06:30:21.048679 (+ 96us) server_negotiation.cc:407] Connection header received
0806 06:30:21.048799 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:30:21.048802 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:30:21.048817 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:30:21.048875 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 06:30:21.049080 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:21.050214 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:21.051508 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:21.052203 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:21.052264 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:30:21.052313 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0806 06:30:21.052510 (+ 197us) server_negotiation.cc:299] Negotiation successful
0806 06:30:21.052554 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":77,"threads_started":1}
I20260806 06:30:21.052659 18160 tablet_service.cc:1467] Processing CreateTablet for tablet bb5d608f66624475a1ce51a4c81dcb8b (DEFAULT_TABLE table=test-table [id=a6e87454765e4d74a715740cf061a08a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 06:30:21.052800 18276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:21.048140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.129:40075 (local address 127.0.0.1:55726)
0806 06:30:21.048579 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0806 06:30:21.048589 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0806 06:30:21.048709 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:30:21.048957 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:30:21.048958 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:30:21.048960 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 06:30:21.049000 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:30:21.049001 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:30:21.050331 (+ 1330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:30:21.050332 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:30:21.051430 (+ 1098us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:30:21.051433 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:30:21.052342 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:30:21.052343 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:30:21.052397 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:30:21.052399 (+ 2us) client_negotiation.cc:769] Sending connection context
0806 06:30:21.052440 (+ 41us) client_negotiation.cc:240] Negotiation successful
0806 06:30:21.052470 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":426,"mutex_wait_us":22,"thread_start_us":71,"threads_started":1}
I20260806 06:30:21.053018 18160 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb5d608f66624475a1ce51a4c81dcb8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 06:30:21.053403 17782 tablet_service.cc:1467] Processing CreateTablet for tablet bb5d608f66624475a1ce51a4c81dcb8b (DEFAULT_TABLE table=test-table [id=a6e87454765e4d74a715740cf061a08a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 06:30:21.053674 17782 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb5d608f66624475a1ce51a4c81dcb8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 06:30:21.055675 18286 tablet_bootstrap.cc:492] T bb5d608f66624475a1ce51a4c81dcb8b P 1d4c54a433a34e7d91d2176fac140297: Bootstrap starting.
I20260806 06:30:21.056227 18287 tablet_bootstrap.cc:492] T bb5d608f66624475a1ce51a4c81dcb8b P bbcbea95d2204210bed7a36961ec71af: Bootstrap starting.
I20260806 06:30:21.056216 17986 tablet_service.cc:1467] Processing CreateTablet for tablet bb5d608f66624475a1ce51a4c81dcb8b (DEFAULT_TABLE table=test-table [id=a6e87454765e4d74a715740cf061a08a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 06:30:21.056394 18286 tablet_bootstrap.cc:654] T bb5d608f66624475a1ce51a4c81dcb8b P 1d4c54a433a34e7d91d2176fac140297: Neither blocks nor log segments found. Creating new log.
I20260806 06:30:21.056543 17986 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb5d608f66624475a1ce51a4c81dcb8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 06:30:21.056844 18286 log.cc:824] T bb5d608f66624475a1ce51a4c81dcb8b P 1d4c54a433a34e7d91d2176fac140297: Log is configured to *not* fsync() on all Append() calls
I20260806 06:30:21.056859 18287 tablet_bootstrap.cc:654] T bb5d608f66624475a1ce51a4c81dcb8b P bbcbea95d2204210bed7a36961ec71af: Neither blocks nor log segments found. Creating new log.
I20260806 06:30:21.057368 18287 log.cc:824] T bb5d608f66624475a1ce51a4c81dcb8b P bbcbea95d2204210bed7a36961ec71af: Log is configured to *not* fsync() on all Append() calls
I20260806 06:30:21.057454 18286 tablet_bootstrap.cc:492] T bb5d608f66624475a1ce51a4c81dcb8b P 1d4c54a433a34e7d91d2176fac140297: No bootstrap required, opened a new log
I20260806 06:30:21.057500 18286 ts_tablet_manager.cc:1397] T bb5d608f66624475a1ce51a4c81dcb8b P 1d4c54a433a34e7d91d2176fac140297: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260806 06:30:21.058125 18287 tablet_bootstrap.cc:492] T bb5d608f66624475a1ce51a4c81dcb8b P bbcbea95d2204210bed7a36961ec71af: No bootstrap required, opened a new log
I20260806 06:30:21.058184 18287 ts_tablet_manager.cc:1397] T bb5d608f66624475a1ce51a4c81dcb8b P bbcbea95d2204210bed7a36961ec71af: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260806 06:30:21.059196 18286 raft_consensus.cc:348] T bb5d608f66624475a1ce51a4c81dcb8b P 1d4c54a433a34e7d91d2176fac140297 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae7a64cd5c614e969183aca131d2962a" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 38183 } } peers { permanent_uuid: "1d4c54a433a34e7d91d2176fac140297" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 45381 } } peers { permanent_uuid: "bbcbea95d2204210bed7a36961ec71af" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 40075 } }
I20260806 06:30:21.059369 18286 raft_consensus.cc:374] T bb5d608f66624475a1ce51a4c81dcb8b P 1d4c54a433a34e7d91d2176fac140297 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 06:30:21.059412 18286 raft_consensus.cc:729] T bb5d608f66624475a1ce51a4c81dcb8b P 1d4c54a433a34e7d91d2176fac140297 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d4c54a433a34e7d91d2176fac140297, State: Initialized, Role: FOLLOWER
I20260806 06:30:21.059541 18286 consensus_queue.cc:260] T bb5d608f66624475a1ce51a4c81dcb8b P 1d4c54a433a34e7d91d2176fac140297 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae7a64cd5c614e969183aca131d2962a" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 38183 } } peers { permanent_uuid: "1d4c54a433a34e7d91d2176fac140297" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 45381 } } peers { permanent_uuid: "bbcbea95d2204210bed7a36961ec71af" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 40075 } }
I20260806 06:30:21.059676 18290 tablet_bootstrap.cc:492] T bb5d608f66624475a1ce51a4c81dcb8b P ae7a64cd5c614e969183aca131d2962a: Bootstrap starting.
I20260806 06:30:21.059849 18286 ts_tablet_manager.cc:1428] T bb5d608f66624475a1ce51a4c81dcb8b P 1d4c54a433a34e7d91d2176fac140297: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260806 06:30:21.059993 18266 heartbeater.cc:503] Master 127.21.12.190:36137 was elected leader, sending a full tablet report...
I20260806 06:30:21.060418 18290 tablet_bootstrap.cc:654] T bb5d608f66624475a1ce51a4c81dcb8b P ae7a64cd5c614e969183aca131d2962a: Neither blocks nor log segments found. Creating new log.
I20260806 06:30:21.060385 18287 raft_consensus.cc:348] T bb5d608f66624475a1ce51a4c81dcb8b P bbcbea95d2204210bed7a36961ec71af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae7a64cd5c614e969183aca131d2962a" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 38183 } } peers { permanent_uuid: "1d4c54a433a34e7d91d2176fac140297" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 45381 } } peers { permanent_uuid: "bbcbea95d2204210bed7a36961ec71af" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 40075 } }
I20260806 06:30:21.060546 18287 raft_consensus.cc:374] T bb5d608f66624475a1ce51a4c81dcb8b P bbcbea95d2204210bed7a36961ec71af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 06:30:21.060583 18287 raft_consensus.cc:729] T bb5d608f66624475a1ce51a4c81dcb8b P bbcbea95d2204210bed7a36961ec71af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbcbea95d2204210bed7a36961ec71af, State: Initialized, Role: FOLLOWER
I20260806 06:30:21.060757 18287 consensus_queue.cc:260] T bb5d608f66624475a1ce51a4c81dcb8b P bbcbea95d2204210bed7a36961ec71af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae7a64cd5c614e969183aca131d2962a" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 38183 } } peers { permanent_uuid: "1d4c54a433a34e7d91d2176fac140297" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 45381 } } peers { permanent_uuid: "bbcbea95d2204210bed7a36961ec71af" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 40075 } }
I20260806 06:30:21.061080 17877 heartbeater.cc:503] Master 127.21.12.190:36137 was elected leader, sending a full tablet report...
I20260806 06:30:21.061182 18290 log.cc:824] T bb5d608f66624475a1ce51a4c81dcb8b P ae7a64cd5c614e969183aca131d2962a: Log is configured to *not* fsync() on all Append() calls
I20260806 06:30:21.061203 18287 ts_tablet_manager.cc:1428] T bb5d608f66624475a1ce51a4c81dcb8b P bbcbea95d2204210bed7a36961ec71af: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260806 06:30:21.061961 18290 tablet_bootstrap.cc:492] T bb5d608f66624475a1ce51a4c81dcb8b P ae7a64cd5c614e969183aca131d2962a: No bootstrap required, opened a new log
I20260806 06:30:21.062034 18290 ts_tablet_manager.cc:1397] T bb5d608f66624475a1ce51a4c81dcb8b P ae7a64cd5c614e969183aca131d2962a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260806 06:30:21.063392 18290 raft_consensus.cc:348] T bb5d608f66624475a1ce51a4c81dcb8b P ae7a64cd5c614e969183aca131d2962a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae7a64cd5c614e969183aca131d2962a" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 38183 } } peers { permanent_uuid: "1d4c54a433a34e7d91d2176fac140297" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 45381 } } peers { permanent_uuid: "bbcbea95d2204210bed7a36961ec71af" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 40075 } }
I20260806 06:30:21.063520 18290 raft_consensus.cc:374] T bb5d608f66624475a1ce51a4c81dcb8b P ae7a64cd5c614e969183aca131d2962a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 06:30:21.063541 18290 raft_consensus.cc:729] T bb5d608f66624475a1ce51a4c81dcb8b P ae7a64cd5c614e969183aca131d2962a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae7a64cd5c614e969183aca131d2962a, State: Initialized, Role: FOLLOWER
I20260806 06:30:21.063616 18290 consensus_queue.cc:260] T bb5d608f66624475a1ce51a4c81dcb8b P ae7a64cd5c614e969183aca131d2962a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae7a64cd5c614e969183aca131d2962a" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 38183 } } peers { permanent_uuid: "1d4c54a433a34e7d91d2176fac140297" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 45381 } } peers { permanent_uuid: "bbcbea95d2204210bed7a36961ec71af" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 40075 } }
I20260806 06:30:21.063814 18290 ts_tablet_manager.cc:1428] T bb5d608f66624475a1ce51a4c81dcb8b P ae7a64cd5c614e969183aca131d2962a: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260806 06:30:21.063918 18071 heartbeater.cc:503] Master 127.21.12.190:36137 was elected leader, sending a full tablet report...
W20260806 06:30:21.133205 18072 tablet.cc:2369] T bb5d608f66624475a1ce51a4c81dcb8b P ae7a64cd5c614e969183aca131d2962a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 06:30:21.201220 18294 raft_consensus.cc:482] T bb5d608f66624475a1ce51a4c81dcb8b P ae7a64cd5c614e969183aca131d2962a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 06:30:21.201351 18294 raft_consensus.cc:504] T bb5d608f66624475a1ce51a4c81dcb8b P ae7a64cd5c614e969183aca131d2962a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae7a64cd5c614e969183aca131d2962a" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 38183 } } peers { permanent_uuid: "1d4c54a433a34e7d91d2176fac140297" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 45381 } } peers { permanent_uuid: "bbcbea95d2204210bed7a36961ec71af" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 40075 } }
I20260806 06:30:21.201771 18294 leader_election.cc:288] T bb5d608f66624475a1ce51a4c81dcb8b P ae7a64cd5c614e969183aca131d2962a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1d4c54a433a34e7d91d2176fac140297 (127.21.12.131:45381), bbcbea95d2204210bed7a36961ec71af (127.21.12.129:40075)
I20260806 06:30:21.204452 18280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:21.202006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.130:48979 (local address 127.21.12.131:45381)
0806 06:30:21.202063 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0806 06:30:21.202064 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0806 06:30:21.202073 (+ 9us) server_negotiation.cc:407] Connection header received
0806 06:30:21.202138 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:30:21.202139 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:30:21.202149 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:30:21.202195 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 06:30:21.202351 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:21.202883 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:21.203795 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:21.204151 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:21.204214 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:30:21.204224 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0806 06:30:21.204357 (+ 133us) server_negotiation.cc:299] Negotiation successful
0806 06:30:21.204389 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260806 06:30:21.204452 18075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:21.201928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.131:45381 (local address 127.21.12.130:48979)
0806 06:30:21.201993 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0806 06:30:21.202005 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 06:30:21.202062 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:30:21.202207 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:30:21.202207 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:30:21.202209 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 06:30:21.202278 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:30:21.202280 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:30:21.202952 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:30:21.202953 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:30:21.203705 (+ 752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:30:21.203707 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:30:21.204218 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:30:21.204219 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:30:21.204294 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:30:21.204297 (+ 3us) client_negotiation.cc:769] Sending connection context
0806 06:30:21.204341 (+ 44us) client_negotiation.cc:240] Negotiation successful
0806 06:30:21.204382 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260806 06:30:21.204685 18284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:21.202048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.130:38017 (local address 127.21.12.129:40075)
0806 06:30:21.202099 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0806 06:30:21.202101 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:30:21.202205 (+ 104us) server_negotiation.cc:407] Connection header received
0806 06:30:21.202264 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:30:21.202265 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:30:21.202276 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:30:21.202357 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 06:30:21.202532 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:21.203016 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:21.203939 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:21.204324 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:21.204395 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:30:21.204403 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0806 06:30:21.204572 (+ 169us) server_negotiation.cc:299] Negotiation successful
0806 06:30:21.204619 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260806 06:30:21.204763 18194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb5d608f66624475a1ce51a4c81dcb8b" candidate_uuid: "ae7a64cd5c614e969183aca131d2962a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d4c54a433a34e7d91d2176fac140297" is_pre_election: true
I20260806 06:30:21.204892 18295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:21.201944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.129:40075 (local address 127.21.12.130:38017)
0806 06:30:21.202130 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0806 06:30:21.202140 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0806 06:30:21.202203 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:30:21.202362 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:30:21.202363 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:30:21.202365 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 06:30:21.202442 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:30:21.202443 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:30:21.203091 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:30:21.203092 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:30:21.203846 (+ 754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:30:21.203848 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:30:21.204398 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:30:21.204399 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:30:21.204474 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:30:21.204478 (+ 4us) client_negotiation.cc:769] Sending connection context
0806 06:30:21.204772 (+ 294us) client_negotiation.cc:240] Negotiation successful
0806 06:30:21.204813 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":114,"threads_started":1}
I20260806 06:30:21.204933 18194 raft_consensus.cc:2454] T bb5d608f66624475a1ce51a4c81dcb8b P 1d4c54a433a34e7d91d2176fac140297 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae7a64cd5c614e969183aca131d2962a in term 0.
I20260806 06:30:21.205199 17905 leader_election.cc:302] T bb5d608f66624475a1ce51a4c81dcb8b P ae7a64cd5c614e969183aca131d2962a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d4c54a433a34e7d91d2176fac140297, ae7a64cd5c614e969183aca131d2962a; no voters:
I20260806 06:30:21.205237 17807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb5d608f66624475a1ce51a4c81dcb8b" candidate_uuid: "ae7a64cd5c614e969183aca131d2962a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbcbea95d2204210bed7a36961ec71af" is_pre_election: true
I20260806 06:30:21.205328 18294 raft_consensus.cc:2781] T bb5d608f66624475a1ce51a4c81dcb8b P ae7a64cd5c614e969183aca131d2962a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 06:30:21.205385 18294 raft_consensus.cc:482] T bb5d608f66624475a1ce51a4c81dcb8b P ae7a64cd5c614e969183aca131d2962a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 06:30:21.205412 18294 raft_consensus.cc:3037] T bb5d608f66624475a1ce51a4c81dcb8b P ae7a64cd5c614e969183aca131d2962a [term 0 FOLLOWER]: Advancing to term 1
I20260806 06:30:21.205403 17807 raft_consensus.cc:2454] T bb5d608f66624475a1ce51a4c81dcb8b P bbcbea95d2204210bed7a36961ec71af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae7a64cd5c614e969183aca131d2962a in term 0.
I20260806 06:30:21.206115 18294 raft_consensus.cc:504] T bb5d608f66624475a1ce51a4c81dcb8b P ae7a64cd5c614e969183aca131d2962a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae7a64cd5c614e969183aca131d2962a" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 38183 } } peers { permanent_uuid: "1d4c54a433a34e7d91d2176fac140297" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 45381 } } peers { permanent_uuid: "bbcbea95d2204210bed7a36961ec71af" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 40075 } }
I20260806 06:30:21.206238 18294 leader_election.cc:288] T bb5d608f66624475a1ce51a4c81dcb8b P ae7a64cd5c614e969183aca131d2962a [CANDIDATE]: Term 1 election: Requested vote from peers 1d4c54a433a34e7d91d2176fac140297 (127.21.12.131:45381), bbcbea95d2204210bed7a36961ec71af (127.21.12.129:40075)
I20260806 06:30:21.206380 17807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb5d608f66624475a1ce51a4c81dcb8b" candidate_uuid: "ae7a64cd5c614e969183aca131d2962a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbcbea95d2204210bed7a36961ec71af"
I20260806 06:30:21.206404 18194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb5d608f66624475a1ce51a4c81dcb8b" candidate_uuid: "ae7a64cd5c614e969183aca131d2962a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d4c54a433a34e7d91d2176fac140297"
I20260806 06:30:21.206476 18194 raft_consensus.cc:3037] T bb5d608f66624475a1ce51a4c81dcb8b P 1d4c54a433a34e7d91d2176fac140297 [term 0 FOLLOWER]: Advancing to term 1
I20260806 06:30:21.206492 17807 raft_consensus.cc:3037] T bb5d608f66624475a1ce51a4c81dcb8b P bbcbea95d2204210bed7a36961ec71af [term 0 FOLLOWER]: Advancing to term 1
I20260806 06:30:21.207232 18194 raft_consensus.cc:2454] T bb5d608f66624475a1ce51a4c81dcb8b P 1d4c54a433a34e7d91d2176fac140297 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae7a64cd5c614e969183aca131d2962a in term 1.
I20260806 06:30:21.207340 17807 raft_consensus.cc:2454] T bb5d608f66624475a1ce51a4c81dcb8b P bbcbea95d2204210bed7a36961ec71af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae7a64cd5c614e969183aca131d2962a in term 1.
I20260806 06:30:21.207370 17905 leader_election.cc:302] T bb5d608f66624475a1ce51a4c81dcb8b P ae7a64cd5c614e969183aca131d2962a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d4c54a433a34e7d91d2176fac140297, ae7a64cd5c614e969183aca131d2962a; no voters:
I20260806 06:30:21.207482 18294 raft_consensus.cc:2781] T bb5d608f66624475a1ce51a4c81dcb8b P ae7a64cd5c614e969183aca131d2962a [term 1 FOLLOWER]: Leader election won for term 1
I20260806 06:30:21.207613 18294 raft_consensus.cc:686] T bb5d608f66624475a1ce51a4c81dcb8b P ae7a64cd5c614e969183aca131d2962a [term 1 LEADER]: Becoming Leader. State: Replica: ae7a64cd5c614e969183aca131d2962a, State: Running, Role: LEADER
I20260806 06:30:21.207686 18294 consensus_queue.cc:237] T bb5d608f66624475a1ce51a4c81dcb8b P ae7a64cd5c614e969183aca131d2962a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae7a64cd5c614e969183aca131d2962a" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 38183 } } peers { permanent_uuid: "1d4c54a433a34e7d91d2176fac140297" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 45381 } } peers { permanent_uuid: "bbcbea95d2204210bed7a36961ec71af" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 40075 } }
I20260806 06:30:21.208446 17589 catalog_manager.cc:5495] T bb5d608f66624475a1ce51a4c81dcb8b P ae7a64cd5c614e969183aca131d2962a reported cstate change: term changed from 0 to 1, leader changed from <none> to ae7a64cd5c614e969183aca131d2962a (127.21.12.130). New cstate: current_term: 1 leader_uuid: "ae7a64cd5c614e969183aca131d2962a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae7a64cd5c614e969183aca131d2962a" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 38183 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1d4c54a433a34e7d91d2176fac140297" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 45381 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bbcbea95d2204210bed7a36961ec71af" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 40075 } health_report { overall_health: UNKNOWN } } }
W20260806 06:30:21.220220 17878 tablet.cc:2369] T bb5d608f66624475a1ce51a4c81dcb8b P bbcbea95d2204210bed7a36961ec71af: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 06:30:21.233476 18282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:21.231258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60838 (local address 127.21.12.130:38183)
0806 06:30:21.231317 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0806 06:30:21.231319 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:30:21.231327 (+ 8us) server_negotiation.cc:407] Connection header received
0806 06:30:21.231360 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:30:21.231361 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:30:21.231380 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:30:21.231438 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 06:30:21.231589 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:21.232049 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:21.232778 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:21.232982 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:21.233042 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:30:21.233182 (+ 140us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 06:30:21.233283 (+ 101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 06:30:21.233350 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0806 06:30:21.233386 (+ 36us) server_negotiation.cc:299] Negotiation successful
0806 06:30:21.233405 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260806 06:30:21.235212 17807 raft_consensus.cc:1261] T bb5d608f66624475a1ce51a4c81dcb8b P bbcbea95d2204210bed7a36961ec71af [term 1 FOLLOWER]: Refusing update from remote peer ae7a64cd5c614e969183aca131d2962a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 06:30:21.235212 18194 raft_consensus.cc:1261] T bb5d608f66624475a1ce51a4c81dcb8b P 1d4c54a433a34e7d91d2176fac140297 [term 1 FOLLOWER]: Refusing update from remote peer ae7a64cd5c614e969183aca131d2962a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 06:30:21.235553 18296 consensus_queue.cc:1035] T bb5d608f66624475a1ce51a4c81dcb8b P ae7a64cd5c614e969183aca131d2962a [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbcbea95d2204210bed7a36961ec71af" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 40075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 06:30:21.235675 18294 consensus_queue.cc:1035] T bb5d608f66624475a1ce51a4c81dcb8b P ae7a64cd5c614e969183aca131d2962a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d4c54a433a34e7d91d2176fac140297" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 45381 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 06:30:21.238286 18306 mvcc.cc:205] Tried to move back new op lower bound from 7315447075776921600 to 7315447075667181568. Current Snapshot: MvccSnapshot[applied={T|T < 7315447075776921600}]
I20260806 06:30:21.238415 18301 mvcc.cc:205] Tried to move back new op lower bound from 7315447075776921600 to 7315447075667181568. Current Snapshot: MvccSnapshot[applied={T|T < 7315447075776921600}]
I20260806 06:30:21.240173 17588 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33048:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 06:30:21.240271 17588 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33048:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 06:30:21.240973 17588 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2f570376f25a452b883cb6a817b4ce26: Sending DeleteTablet for 3 replicas of tablet bb5d608f66624475a1ce51a4c81dcb8b
I20260806 06:30:21.241297 17986 tablet_service.cc:1514] Processing DeleteTablet for tablet bb5d608f66624475a1ce51a4c81dcb8b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 06:30:21 UTC) from {username='kudu', principal='kudu/127.21.12.190@KRBTEST.COM'} at 127.0.0.1:60836
I20260806 06:30:21.241328 17782 tablet_service.cc:1514] Processing DeleteTablet for tablet bb5d608f66624475a1ce51a4c81dcb8b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 06:30:21 UTC) from {username='kudu', principal='kudu/127.21.12.190@KRBTEST.COM'} at 127.0.0.1:55726
I20260806 06:30:21.241349 18160 tablet_service.cc:1514] Processing DeleteTablet for tablet bb5d608f66624475a1ce51a4c81dcb8b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 06:30:21 UTC) from {username='kudu', principal='kudu/127.21.12.190@KRBTEST.COM'} at 127.0.0.1:35726
I20260806 06:30:21.241542 18317 tablet_replica.cc:330] T bb5d608f66624475a1ce51a4c81dcb8b P ae7a64cd5c614e969183aca131d2962a: stopping tablet replica
I20260806 06:30:21.241562 18318 tablet_replica.cc:330] T bb5d608f66624475a1ce51a4c81dcb8b P bbcbea95d2204210bed7a36961ec71af: stopping tablet replica
I20260806 06:30:21.241642 18317 raft_consensus.cc:2229] T bb5d608f66624475a1ce51a4c81dcb8b P ae7a64cd5c614e969183aca131d2962a [term 1 LEADER]: Raft consensus shutting down.
I20260806 06:30:21.241652 18318 raft_consensus.cc:2229] T bb5d608f66624475a1ce51a4c81dcb8b P bbcbea95d2204210bed7a36961ec71af [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 06:30:21.241737 18317 raft_consensus.cc:2258] T bb5d608f66624475a1ce51a4c81dcb8b P ae7a64cd5c614e969183aca131d2962a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 06:30:21.241743 18318 raft_consensus.cc:2258] T bb5d608f66624475a1ce51a4c81dcb8b P bbcbea95d2204210bed7a36961ec71af [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 06:30:21.241734 21554 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHhLhT/build/release/bin/kudu with pid 17689
I20260806 06:30:21.251430 18317 ts_tablet_manager.cc:1905] T bb5d608f66624475a1ce51a4c81dcb8b P ae7a64cd5c614e969183aca131d2962a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 06:30:21.253101 18317 ts_tablet_manager.cc:1918] T bb5d608f66624475a1ce51a4c81dcb8b P ae7a64cd5c614e969183aca131d2962a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 06:30:21.253181 18317 log.cc:1196] T bb5d608f66624475a1ce51a4c81dcb8b P ae7a64cd5c614e969183aca131d2962a: Deleting WAL directory at /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785997581049705-21554-0/minicluster-data/ts-1/wal/wals/bb5d608f66624475a1ce51a4c81dcb8b
W20260806 06:30:21.253394 17570 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.12.129:40075 (error 108)
I20260806 06:30:21.253511 18317 ts_tablet_manager.cc:1939] T bb5d608f66624475a1ce51a4c81dcb8b P ae7a64cd5c614e969183aca131d2962a: Deleting consensus metadata
I20260806 06:30:21.253902 17568 catalog_manager.cc:4841] TS ae7a64cd5c614e969183aca131d2962a (127.21.12.130:38183): tablet bb5d608f66624475a1ce51a4c81dcb8b (table test-table [id=a6e87454765e4d74a715740cf061a08a]) successfully deleted
I20260806 06:30:21.254174 18276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:21.254049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.129:40075 (local address 127.0.0.1:55730)
0806 06:30:21.254094 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0806 06:30:21.254140 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.129:40075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
I20260806 06:30:21.254216 18319 tablet_replica.cc:330] T bb5d608f66624475a1ce51a4c81dcb8b P 1d4c54a433a34e7d91d2176fac140297: stopping tablet replica
I20260806 06:30:21.254209 21554 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHhLhT/build/release/bin/kudu with pid 17883
W20260806 06:30:21.254285 17570 catalog_manager.cc:4584] TS bbcbea95d2204210bed7a36961ec71af (127.21.12.129:40075): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet bb5d608f66624475a1ce51a4c81dcb8b: Network error: Client connection negotiation failed: client connection to 127.21.12.129:40075: connect: Connection refused (error 111)
I20260806 06:30:21.254359 18319 raft_consensus.cc:2229] T bb5d608f66624475a1ce51a4c81dcb8b P 1d4c54a433a34e7d91d2176fac140297 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 06:30:21.254468 18319 raft_consensus.cc:2258] T bb5d608f66624475a1ce51a4c81dcb8b P 1d4c54a433a34e7d91d2176fac140297 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 06:30:21.256045 18319 ts_tablet_manager.cc:1905] T bb5d608f66624475a1ce51a4c81dcb8b P 1d4c54a433a34e7d91d2176fac140297: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 06:30:21.257364 18319 ts_tablet_manager.cc:1918] T bb5d608f66624475a1ce51a4c81dcb8b P 1d4c54a433a34e7d91d2176fac140297: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 06:30:21.257439 18319 log.cc:1196] T bb5d608f66624475a1ce51a4c81dcb8b P 1d4c54a433a34e7d91d2176fac140297: Deleting WAL directory at /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785997581049705-21554-0/minicluster-data/ts-2/wal/wals/bb5d608f66624475a1ce51a4c81dcb8b
I20260806 06:30:21.257742 18319 ts_tablet_manager.cc:1939] T bb5d608f66624475a1ce51a4c81dcb8b P 1d4c54a433a34e7d91d2176fac140297: Deleting consensus metadata
I20260806 06:30:21.258217 17567 catalog_manager.cc:4841] TS 1d4c54a433a34e7d91d2176fac140297 (127.21.12.131:45381): tablet bb5d608f66624475a1ce51a4c81dcb8b (table test-table [id=a6e87454765e4d74a715740cf061a08a]) successfully deleted
I20260806 06:30:21.264336 21554 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHhLhT/build/release/bin/kudu with pid 18080
I20260806 06:30:21.273466 21554 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHhLhT/build/release/bin/kudu with pid 17541
2026-08-06T06:30:21Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2075 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-08-06T06:30:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T06:30:21Z Disabled control of system clock
I20260806 06:30:21.302767 21554 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHhLhT/build/release/bin/kudu
/tmp/dist-test-taskFHhLhT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785997581049705-21554-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785997581049705-21554-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785997581049705-21554-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785997581049705-21554-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.12.190:41463
--webserver_interface=127.21.12.190
--webserver_port=0
--builtin_ntp_servers=127.21.12.148:38875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.12.190:41463
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260806 06:30:21.382073 18329 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 06:30:21.382274 18329 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 06:30:21.382316 18329 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 06:30:21.383394 18329 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 06:30:21.383424 18329 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 06:30:21.383436 18329 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 06:30:21.383447 18329 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 06:30:21.383460 18329 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 06:30:21.384733 18329 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.148:38875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785997581049705-21554-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785997581049705-21554-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.12.190:41463
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.12.190:41463
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785997581049705-21554-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.12.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18329
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785997581049705-21554-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 05:11:02 UTC on ccycloud.quasar-hvzbbz.root.comops.site
build id 4569617
I20260806 06:30:21.384909 18329 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 06:30:21.385150 18329 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 06:30:21.389807 18336 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:30:21.389866 18335 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:30:21.389879 18329 server_base.cc:1034] running on GCE node
W20260806 06:30:21.390046 18338 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:30:21.390272 18329 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 06:30:21.390594 18329 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 06:30:21.391753 18329 hybrid_clock.cc:648] HybridClock initialized: now 1785997821391744 us; error 29 us; skew 500 ppm
W20260806 06:30:22.039870 18329 net_util.cc:504] Time spent look up canonical hostname for localhost 'dist-test-slave-4tdj': real 0.648s user 0.000s sys 0.001s
I20260806 06:30:22.040582 18329 webserver.cc:460] Webserver started at http://127.21.12.190:35301/ using document root <none> and password file <none>
I20260806 06:30:22.040903 18329 fs_manager.cc:359] Metadata directory not provided
I20260806 06:30:22.040972 18329 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 06:30:22.041139 18329 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 06:30:22.042572 18329 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785997581049705-21554-0/minicluster-data/master-0/data/instance:
uuid: "0e835e0f83014cce8e5cd1832af65fb8"
format_stamp: "Formatted at 2026-08-06 06:30:22 on dist-test-slave-4tdj"
server_key: "515170f7f60c91c4d628804e729745d7"
server_key_iv: "df9494bc1d21b60af3e4d9b27510f3eb"
server_key_version: "encryptionkey@0"
I20260806 06:30:22.042982 18329 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785997581049705-21554-0/minicluster-data/master-0/wal/instance:
uuid: "0e835e0f83014cce8e5cd1832af65fb8"
format_stamp: "Formatted at 2026-08-06 06:30:22 on dist-test-slave-4tdj"
server_key: "515170f7f60c91c4d628804e729745d7"
server_key_iv: "df9494bc1d21b60af3e4d9b27510f3eb"
server_key_version: "encryptionkey@0"
I20260806 06:30:22.044759 18329 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.002s
I20260806 06:30:22.045660 18351 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 06:30:22.045873 18329 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260806 06:30:22.045977 18329 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785997581049705-21554-0/minicluster-data/master-0/data,/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785997581049705-21554-0/minicluster-data/master-0/wal
uuid: "0e835e0f83014cce8e5cd1832af65fb8"
format_stamp: "Formatted at 2026-08-06 06:30:22 on dist-test-slave-4tdj"
server_key: "515170f7f60c91c4d628804e729745d7"
server_key_iv: "df9494bc1d21b60af3e4d9b27510f3eb"
server_key_version: "encryptionkey@0"
I20260806 06:30:22.046064 18329 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785997581049705-21554-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785997581049705-21554-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785997581049705-21554-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 06:30:22.063150 18329 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 06:30:22.063544 18329 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 06:30:22.063683 18329 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 06:30:22.069384 18329 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.190:41463
I20260806 06:30:22.069415 18422 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.190:41463 every 8 connection(s)
I20260806 06:30:22.069797 18329 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785997581049705-21554-0/minicluster-data/master-0/data/info.pb
I20260806 06:30:22.070959 18425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 06:30:22.074065 18425 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0e835e0f83014cce8e5cd1832af65fb8: Bootstrap starting.
I20260806 06:30:22.074781 18425 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0e835e0f83014cce8e5cd1832af65fb8: Neither blocks nor log segments found. Creating new log.
I20260806 06:30:22.075124 21554 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHhLhT/build/release/bin/kudu as pid 18329
I20260806 06:30:22.075132 18425 log.cc:824] T 00000000000000000000000000000000 P 0e835e0f83014cce8e5cd1832af65fb8: Log is configured to *not* fsync() on all Append() calls
I20260806 06:30:22.075229 21554 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785997581049705-21554-0/minicluster-data/master-0/wal/instance
I20260806 06:30:22.075420 21554 external_mini_cluster.cc:1428] Setting key 7b7b5adddc26bbeefc02aa6458bd6ffd
I20260806 06:30:22.075860 18425 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0e835e0f83014cce8e5cd1832af65fb8: No bootstrap required, opened a new log
I20260806 06:30:22.077579 18425 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0e835e0f83014cce8e5cd1832af65fb8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e835e0f83014cce8e5cd1832af65fb8" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 41463 } }
I20260806 06:30:22.077718 18425 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0e835e0f83014cce8e5cd1832af65fb8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 06:30:22.077749 18425 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0e835e0f83014cce8e5cd1832af65fb8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e835e0f83014cce8e5cd1832af65fb8, State: Initialized, Role: FOLLOWER
I20260806 06:30:22.077873 18425 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0e835e0f83014cce8e5cd1832af65fb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e835e0f83014cce8e5cd1832af65fb8" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 41463 } }
I20260806 06:30:22.077950 18425 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0e835e0f83014cce8e5cd1832af65fb8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 06:30:22.077986 18425 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0e835e0f83014cce8e5cd1832af65fb8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 06:30:22.078025 18425 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0e835e0f83014cce8e5cd1832af65fb8 [term 0 FOLLOWER]: Advancing to term 1
I20260806 06:30:22.078727 18425 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0e835e0f83014cce8e5cd1832af65fb8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e835e0f83014cce8e5cd1832af65fb8" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 41463 } }
I20260806 06:30:22.078835 18425 leader_election.cc:302] T 00000000000000000000000000000000 P 0e835e0f83014cce8e5cd1832af65fb8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0e835e0f83014cce8e5cd1832af65fb8; no voters:
I20260806 06:30:22.078999 18425 leader_election.cc:288] T 00000000000000000000000000000000 P 0e835e0f83014cce8e5cd1832af65fb8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 06:30:22.079071 18430 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0e835e0f83014cce8e5cd1832af65fb8 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 06:30:22.079224 18430 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0e835e0f83014cce8e5cd1832af65fb8 [term 1 LEADER]: Becoming Leader. State: Replica: 0e835e0f83014cce8e5cd1832af65fb8, State: Running, Role: LEADER
I20260806 06:30:22.079260 18425 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0e835e0f83014cce8e5cd1832af65fb8 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 06:30:22.079319 18430 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0e835e0f83014cce8e5cd1832af65fb8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e835e0f83014cce8e5cd1832af65fb8" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 41463 } }
I20260806 06:30:22.079670 18430 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e835e0f83014cce8e5cd1832af65fb8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0e835e0f83014cce8e5cd1832af65fb8. Latest consensus state: current_term: 1 leader_uuid: "0e835e0f83014cce8e5cd1832af65fb8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e835e0f83014cce8e5cd1832af65fb8" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 41463 } } }
I20260806 06:30:22.079746 18430 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e835e0f83014cce8e5cd1832af65fb8 [sys.catalog]: This master's current role is: LEADER
I20260806 06:30:22.079897 18432 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e835e0f83014cce8e5cd1832af65fb8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0e835e0f83014cce8e5cd1832af65fb8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e835e0f83014cce8e5cd1832af65fb8" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 41463 } } }
I20260806 06:30:22.079962 18432 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e835e0f83014cce8e5cd1832af65fb8 [sys.catalog]: This master's current role is: LEADER
I20260806 06:30:22.080319 18441 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 06:30:22.080847 18441 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 06:30:22.083032 18441 catalog_manager.cc:1347] Generated new cluster ID: cba2b5fc9432433f8d400344e0b0660d
I20260806 06:30:22.083101 18441 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 06:30:22.087918 18429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:22.075905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57654 (local address 127.21.12.190:41463)
0806 06:30:22.076096 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0806 06:30:22.076106 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 06:30:22.076142 (+ 36us) server_negotiation.cc:407] Connection header received
0806 06:30:22.077567 (+ 1425us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:30:22.077578 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:30:22.077669 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:30:22.077825 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:30:22.078177 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:22.080542 (+ 2365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:22.081633 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:22.082247 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:22.082398 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:30:22.086886 (+ 4488us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:30:22.086890 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:30:22.086891 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 06:30:22.087102 (+ 211us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:30:22.087149 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 06:30:22.087164 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:30:22.087225 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0806 06:30:22.087510 (+ 285us) server_negotiation.cc:299] Negotiation successful
0806 06:30:22.087598 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":69,"threads_started":1}
I20260806 06:30:22.096493 18441 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 06:30:22.097018 18441 catalog_manager.cc:1504] Loading token signing keys...
I20260806 06:30:22.100020 18441 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0e835e0f83014cce8e5cd1832af65fb8: Generated new TSK 0
I20260806 06:30:22.100175 18441 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 06:30:22.106143 21554 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHhLhT/build/release/bin/kudu
/tmp/dist-test-taskFHhLhT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785997581049705-21554-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785997581049705-21554-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785997581049705-21554-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785997581049705-21554-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.12.129:0
--local_ip_for_outbound_sockets=127.21.12.129
--webserver_interface=127.21.12.129
--webserver_port=0
--tserver_master_addrs=127.21.12.190:41463
--builtin_ntp_servers=127.21.12.148:38875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260806 06:30:22.188149 18462 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 06:30:22.188336 18462 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 06:30:22.188356 18462 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 06:30:22.189467 18462 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 06:30:22.189507 18462 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 06:30:22.189530 18462 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.12.129
I20260806 06:30:22.190892 18462 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.148:38875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785997581049705-21554-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785997581049705-21554-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.12.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785997581049705-21554-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.12.129
--webserver_port=0
--tserver_master_addrs=127.21.12.190:41463
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18462
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.12.129
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785997581049705-21554-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 05:11:02 UTC on ccycloud.quasar-hvzbbz.root.comops.site
build id 4569617
I20260806 06:30:22.191129 18462 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 06:30:22.191401 18462 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 06:30:22.197162 18467 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:30:22.197178 18472 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:30:22.197232 18470 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:30:22.197430 18462 server_base.cc:1034] running on GCE node
I20260806 06:30:22.197677 18462 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 06:30:22.197893 18462 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 06:30:22.199028 18462 hybrid_clock.cc:648] HybridClock initialized: now 1785997822199015 us; error 31 us; skew 500 ppm
I20260806 06:30:22.200196 18462 webserver.cc:460] Webserver started at http://127.21.12.129:45435/ using document root <none> and password file <none>
I20260806 06:30:22.200418 18462 fs_manager.cc:359] Metadata directory not provided
I20260806 06:30:22.200466 18462 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 06:30:22.200610 18462 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 06:30:22.201659 18462 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785997581049705-21554-0/minicluster-data/ts-0/data/instance:
uuid: "09e36e01849540a6aa725ca9e4b0f6f9"
format_stamp: "Formatted at 2026-08-06 06:30:22 on dist-test-slave-4tdj"
server_key: "c06a5ab869e38d0a1697c1ab3059c51e"
server_key_iv: "0a740caea7fee0cd5c0c6d69d363ed77"
server_key_version: "encryptionkey@0"
I20260806 06:30:22.201988 18462 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785997581049705-21554-0/minicluster-data/ts-0/wal/instance:
uuid: "09e36e01849540a6aa725ca9e4b0f6f9"
format_stamp: "Formatted at 2026-08-06 06:30:22 on dist-test-slave-4tdj"
server_key: "c06a5ab869e38d0a1697c1ab3059c51e"
server_key_iv: "0a740caea7fee0cd5c0c6d69d363ed77"
server_key_version: "encryptionkey@0"
I20260806 06:30:22.203272 18462 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260806 06:30:22.203995 18479 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 06:30:22.204170 18462 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260806 06:30:22.204272 18462 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785997581049705-21554-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785997581049705-21554-0/minicluster-data/ts-0/wal
uuid: "09e36e01849540a6aa725ca9e4b0f6f9"
format_stamp: "Formatted at 2026-08-06 06:30:22 on dist-test-slave-4tdj"
server_key: "c06a5ab869e38d0a1697c1ab3059c51e"
server_key_iv: "0a740caea7fee0cd5c0c6d69d363ed77"
server_key_version: "encryptionkey@0"
I20260806 06:30:22.204346 18462 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785997581049705-21554-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785997581049705-21554-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785997581049705-21554-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 06:30:22.215271 18462 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 06:30:22.215528 18462 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 06:30:22.215623 18462 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 06:30:22.215792 18462 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 06:30:22.216049 18462 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 06:30:22.216078 18462 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 06:30:22.216104 18462 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 06:30:22.216120 18462 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 06:30:22.221380 18462 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.129:42139
I20260806 06:30:22.221446 18641 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.129:42139 every 8 connection(s)
I20260806 06:30:22.221747 18462 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785997581049705-21554-0/minicluster-data/ts-0/data/info.pb
I20260806 06:30:22.226543 18429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:22.222440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.129:45831 (local address 127.21.12.190:41463)
0806 06:30:22.222520 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0806 06:30:22.222522 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:30:22.222729 (+ 207us) server_negotiation.cc:407] Connection header received
0806 06:30:22.223209 (+ 480us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:30:22.223210 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:30:22.223223 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:30:22.223273 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:30:22.223934 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:22.224427 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:22.225345 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:22.225545 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:22.225593 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:30:22.226267 (+ 674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:30:22.226268 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:30:22.226269 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 06:30:22.226273 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:30:22.226312 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 06:30:22.226319 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:30:22.226369 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0806 06:30:22.226432 (+ 63us) server_negotiation.cc:299] Negotiation successful
0806 06:30:22.226461 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260806 06:30:22.226683 18646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:22.222442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:41463 (local address 127.21.12.129:45831)
0806 06:30:22.222619 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0806 06:30:22.222654 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0806 06:30:22.223125 (+ 471us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:30:22.223327 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:30:22.223330 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:30:22.223338 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0806 06:30:22.223837 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:30:22.223841 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:30:22.224496 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:30:22.224496 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:30:22.225285 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:30:22.225287 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:30:22.225598 (+ 311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:30:22.225599 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:30:22.225699 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:30:22.226150 (+ 451us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 06:30:22.226151 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0806 06:30:22.226177 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 06:30:22.226199 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 06:30:22.226371 (+ 172us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 06:30:22.226374 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 06:30:22.226377 (+ 3us) client_negotiation.cc:769] Sending connection context
0806 06:30:22.226425 (+ 48us) client_negotiation.cc:240] Negotiation successful
0806 06:30:22.226503 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":86,"threads_started":1}
I20260806 06:30:22.227152 18644 heartbeater.cc:347] Connected to a master server at 127.21.12.190:41463
I20260806 06:30:22.227233 18644 heartbeater.cc:464] Registering TS with master...
I20260806 06:30:22.227394 18644 heartbeater.cc:511] Master 127.21.12.190:41463 requested a full tablet report, sending...
I20260806 06:30:22.227851 18373 ts_manager.cc:194] Registered new tserver with Master: 09e36e01849540a6aa725ca9e4b0f6f9 (127.21.12.129:42139)
I20260806 06:30:22.228446 18373 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.12.129:45831
I20260806 06:30:22.231434 21554 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHhLhT/build/release/bin/kudu as pid 18462
I20260806 06:30:22.231530 21554 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785997581049705-21554-0/minicluster-data/ts-0/wal/instance
I20260806 06:30:22.231722 21554 external_mini_cluster.cc:1428] Setting key ea40709243c9a7203cbdeb811a73ef34
I20260806 06:30:22.232601 21554 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHhLhT/build/release/bin/kudu
/tmp/dist-test-taskFHhLhT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785997581049705-21554-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785997581049705-21554-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785997581049705-21554-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785997581049705-21554-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.12.130:0
--local_ip_for_outbound_sockets=127.21.12.130
--webserver_interface=127.21.12.130
--webserver_port=0
--tserver_master_addrs=127.21.12.190:41463
--builtin_ntp_servers=127.21.12.148:38875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260806 06:30:22.312270 18647 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 06:30:22.312471 18647 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 06:30:22.312491 18647 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 06:30:22.313616 18647 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 06:30:22.313655 18647 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 06:30:22.313678 18647 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.12.130
I20260806 06:30:22.315074 18647 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.148:38875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785997581049705-21554-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785997581049705-21554-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.12.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785997581049705-21554-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.12.130
--webserver_port=0
--tserver_master_addrs=127.21.12.190:41463
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18647
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.12.130
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785997581049705-21554-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 05:11:02 UTC on ccycloud.quasar-hvzbbz.root.comops.site
build id 4569617
I20260806 06:30:22.315306 18647 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 06:30:22.315605 18647 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 06:30:22.320443 18653 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:30:22.320510 18655 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:30:22.320464 18658 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:30:22.320750 18647 server_base.cc:1034] running on GCE node
I20260806 06:30:22.321012 18647 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 06:30:22.321233 18647 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 06:30:22.322345 18647 hybrid_clock.cc:648] HybridClock initialized: now 1785997822322325 us; error 37 us; skew 500 ppm
I20260806 06:30:22.323594 18647 webserver.cc:460] Webserver started at http://127.21.12.130:44491/ using document root <none> and password file <none>
I20260806 06:30:22.323840 18647 fs_manager.cc:359] Metadata directory not provided
I20260806 06:30:22.323889 18647 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 06:30:22.324023 18647 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 06:30:22.325073 18647 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785997581049705-21554-0/minicluster-data/ts-1/data/instance:
uuid: "ba8ce598e8bf408bad298fd0ff39238a"
format_stamp: "Formatted at 2026-08-06 06:30:22 on dist-test-slave-4tdj"
server_key: "e8956d718cddef558ec193312436ec32"
server_key_iv: "6762a30b3fad4904f5efea924a6d8caa"
server_key_version: "encryptionkey@0"
I20260806 06:30:22.325423 18647 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785997581049705-21554-0/minicluster-data/ts-1/wal/instance:
uuid: "ba8ce598e8bf408bad298fd0ff39238a"
format_stamp: "Formatted at 2026-08-06 06:30:22 on dist-test-slave-4tdj"
server_key: "e8956d718cddef558ec193312436ec32"
server_key_iv: "6762a30b3fad4904f5efea924a6d8caa"
server_key_version: "encryptionkey@0"
I20260806 06:30:22.326756 18647 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260806 06:30:22.327427 18666 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 06:30:22.327587 18647 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260806 06:30:22.327672 18647 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785997581049705-21554-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785997581049705-21554-0/minicluster-data/ts-1/wal
uuid: "ba8ce598e8bf408bad298fd0ff39238a"
format_stamp: "Formatted at 2026-08-06 06:30:22 on dist-test-slave-4tdj"
server_key: "e8956d718cddef558ec193312436ec32"
server_key_iv: "6762a30b3fad4904f5efea924a6d8caa"
server_key_version: "encryptionkey@0"
I20260806 06:30:22.327737 18647 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785997581049705-21554-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785997581049705-21554-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785997581049705-21554-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 06:30:22.346705 18647 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 06:30:22.347002 18647 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 06:30:22.347127 18647 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 06:30:22.347332 18647 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 06:30:22.347682 18647 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 06:30:22.347719 18647 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 06:30:22.347748 18647 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 06:30:22.347772 18647 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 06:30:22.353349 18647 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.130:41265
I20260806 06:30:22.353389 18838 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.130:41265 every 8 connection(s)
I20260806 06:30:22.353720 18647 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785997581049705-21554-0/minicluster-data/ts-1/data/info.pb
I20260806 06:30:22.357806 21554 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHhLhT/build/release/bin/kudu as pid 18647
I20260806 06:30:22.357896 21554 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785997581049705-21554-0/minicluster-data/ts-1/wal/instance
I20260806 06:30:22.358055 21554 external_mini_cluster.cc:1428] Setting key c2bf475ba6f7c57fa4ebb91b0e1cc618
I20260806 06:30:22.358776 18429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:22.354452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.130:33243 (local address 127.21.12.190:41463)
0806 06:30:22.354502 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0806 06:30:22.354504 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:30:22.354754 (+ 250us) server_negotiation.cc:407] Connection header received
0806 06:30:22.355238 (+ 484us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:30:22.355239 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:30:22.355250 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:30:22.355296 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:30:22.355925 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:22.356405 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:22.357374 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:22.357583 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:22.357636 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:30:22.358440 (+ 804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:30:22.358441 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:30:22.358442 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 06:30:22.358446 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:30:22.358464 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 06:30:22.358468 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:30:22.358522 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0806 06:30:22.358674 (+ 152us) server_negotiation.cc:299] Negotiation successful
0806 06:30:22.358701 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260806 06:30:22.358991 18842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:22.354434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:41463 (local address 127.21.12.130:33243)
0806 06:30:22.354632 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0806 06:30:22.354667 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0806 06:30:22.355151 (+ 484us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:30:22.355359 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:30:22.355362 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:30:22.355371 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0806 06:30:22.355863 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:30:22.355868 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:30:22.356501 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:30:22.356502 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:30:22.357297 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:30:22.357300 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:30:22.357643 (+ 343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:30:22.357644 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:30:22.357746 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:30:22.358272 (+ 526us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 06:30:22.358272 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0806 06:30:22.358328 (+ 56us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 06:30:22.358357 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 06:30:22.358580 (+ 223us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 06:30:22.358587 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 06:30:22.358589 (+ 2us) client_negotiation.cc:769] Sending connection context
0806 06:30:22.358676 (+ 87us) client_negotiation.cc:240] Negotiation successful
0806 06:30:22.358776 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":108,"threads_started":1}
I20260806 06:30:22.359157 21554 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHhLhT/build/release/bin/kudu
/tmp/dist-test-taskFHhLhT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785997581049705-21554-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785997581049705-21554-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785997581049705-21554-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785997581049705-21554-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.12.131:0
--local_ip_for_outbound_sockets=127.21.12.131
--webserver_interface=127.21.12.131
--webserver_port=0
--tserver_master_addrs=127.21.12.190:41463
--builtin_ntp_servers=127.21.12.148:38875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260806 06:30:22.360891 18839 heartbeater.cc:347] Connected to a master server at 127.21.12.190:41463
I20260806 06:30:22.360993 18839 heartbeater.cc:464] Registering TS with master...
I20260806 06:30:22.361166 18839 heartbeater.cc:511] Master 127.21.12.190:41463 requested a full tablet report, sending...
I20260806 06:30:22.361536 18373 ts_manager.cc:194] Registered new tserver with Master: ba8ce598e8bf408bad298fd0ff39238a (127.21.12.130:41265)
I20260806 06:30:22.361961 18373 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.12.130:33243
W20260806 06:30:22.439096 18847 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 06:30:22.439284 18847 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 06:30:22.439303 18847 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 06:30:22.440371 18847 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 06:30:22.440403 18847 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 06:30:22.440425 18847 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.12.131
I20260806 06:30:22.441744 18847 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.148:38875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785997581049705-21554-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785997581049705-21554-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.12.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785997581049705-21554-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.12.131
--webserver_port=0
--tserver_master_addrs=127.21.12.190:41463
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18847
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.12.131
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785997581049705-21554-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 05:11:02 UTC on ccycloud.quasar-hvzbbz.root.comops.site
build id 4569617
I20260806 06:30:22.441920 18847 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 06:30:22.442155 18847 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 06:30:22.446765 18853 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:30:22.446794 18859 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:30:22.446832 18854 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:30:22.446906 18847 server_base.cc:1034] running on GCE node
I20260806 06:30:22.447147 18847 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 06:30:22.447336 18847 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 06:30:22.448482 18847 hybrid_clock.cc:648] HybridClock initialized: now 1785997822448444 us; error 45 us; skew 500 ppm
I20260806 06:30:22.449666 18847 webserver.cc:460] Webserver started at http://127.21.12.131:34029/ using document root <none> and password file <none>
I20260806 06:30:22.449864 18847 fs_manager.cc:359] Metadata directory not provided
I20260806 06:30:22.449910 18847 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 06:30:22.450037 18847 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 06:30:22.451092 18847 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785997581049705-21554-0/minicluster-data/ts-2/data/instance:
uuid: "68f4a0ed90d04d83a9ee1748e4a39064"
format_stamp: "Formatted at 2026-08-06 06:30:22 on dist-test-slave-4tdj"
server_key: "c58c54e228549dd432e6a875cb421873"
server_key_iv: "b5d5c48f5ad245ab92c1f8b46755fc9f"
server_key_version: "encryptionkey@0"
I20260806 06:30:22.451407 18847 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785997581049705-21554-0/minicluster-data/ts-2/wal/instance:
uuid: "68f4a0ed90d04d83a9ee1748e4a39064"
format_stamp: "Formatted at 2026-08-06 06:30:22 on dist-test-slave-4tdj"
server_key: "c58c54e228549dd432e6a875cb421873"
server_key_iv: "b5d5c48f5ad245ab92c1f8b46755fc9f"
server_key_version: "encryptionkey@0"
I20260806 06:30:22.452626 18847 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260806 06:30:22.453282 18867 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 06:30:22.453447 18847 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260806 06:30:22.453557 18847 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785997581049705-21554-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785997581049705-21554-0/minicluster-data/ts-2/wal
uuid: "68f4a0ed90d04d83a9ee1748e4a39064"
format_stamp: "Formatted at 2026-08-06 06:30:22 on dist-test-slave-4tdj"
server_key: "c58c54e228549dd432e6a875cb421873"
server_key_iv: "b5d5c48f5ad245ab92c1f8b46755fc9f"
server_key_version: "encryptionkey@0"
I20260806 06:30:22.453616 18847 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785997581049705-21554-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785997581049705-21554-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785997581049705-21554-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 06:30:22.468303 18847 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 06:30:22.468629 18847 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 06:30:22.468755 18847 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 06:30:22.468948 18847 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 06:30:22.469237 18847 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 06:30:22.469271 18847 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 06:30:22.469300 18847 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 06:30:22.469316 18847 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 06:30:22.474850 18847 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.131:45883
I20260806 06:30:22.474895 19029 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.131:45883 every 8 connection(s)
I20260806 06:30:22.475181 18847 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785997581049705-21554-0/minicluster-data/ts-2/data/info.pb
I20260806 06:30:22.480075 18429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:22.475952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.131:40539 (local address 127.21.12.190:41463)
0806 06:30:22.476025 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0806 06:30:22.476027 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:30:22.476139 (+ 112us) server_negotiation.cc:407] Connection header received
0806 06:30:22.476641 (+ 502us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:30:22.476642 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:30:22.476655 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:30:22.476703 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:30:22.477338 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:22.477834 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:22.478830 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:22.479037 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:22.479120 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:30:22.479860 (+ 740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:30:22.479861 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:30:22.479861 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 06:30:22.479865 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:30:22.479880 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 06:30:22.479883 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:30:22.479922 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0806 06:30:22.479982 (+ 60us) server_negotiation.cc:299] Negotiation successful
0806 06:30:22.480008 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260806 06:30:22.480280 19034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:22.475815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:41463 (local address 127.21.12.131:40539)
0806 06:30:22.476016 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0806 06:30:22.476059 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0806 06:30:22.476521 (+ 462us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:30:22.476782 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:30:22.476785 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:30:22.476793 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0806 06:30:22.477256 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:30:22.477261 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:30:22.477917 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:30:22.477918 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:30:22.478752 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:30:22.478755 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:30:22.479169 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:30:22.479170 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:30:22.479308 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:30:22.479737 (+ 429us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 06:30:22.479737 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0806 06:30:22.479764 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 06:30:22.479785 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 06:30:22.479927 (+ 142us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 06:30:22.479929 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 06:30:22.479932 (+ 3us) client_negotiation.cc:769] Sending connection context
0806 06:30:22.479975 (+ 43us) client_negotiation.cc:240] Negotiation successful
0806 06:30:22.480060 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":93,"threads_started":1}
I20260806 06:30:22.480707 19031 heartbeater.cc:347] Connected to a master server at 127.21.12.190:41463
I20260806 06:30:22.480791 19031 heartbeater.cc:464] Registering TS with master...
I20260806 06:30:22.480955 19031 heartbeater.cc:511] Master 127.21.12.190:41463 requested a full tablet report, sending...
I20260806 06:30:22.481268 18373 ts_manager.cc:194] Registered new tserver with Master: 68f4a0ed90d04d83a9ee1748e4a39064 (127.21.12.131:45883)
I20260806 06:30:22.481683 18373 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.12.131:40539
I20260806 06:30:22.484540 21554 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHhLhT/build/release/bin/kudu as pid 18847
I20260806 06:30:22.484633 21554 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785997581049705-21554-0/minicluster-data/ts-2/wal/instance
I20260806 06:30:22.484773 21554 external_mini_cluster.cc:1428] Setting key efa67ec8027eb7fe18cc825fe1683259
I20260806 06:30:22.485517 21554 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 06:30:22.489356 18429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:22.486780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57668 (local address 127.21.12.190:41463)
0806 06:30:22.486836 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0806 06:30:22.486838 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:30:22.486933 (+ 95us) server_negotiation.cc:407] Connection header received
0806 06:30:22.486994 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:30:22.486995 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:30:22.487006 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:30:22.487062 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:30:22.487209 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:22.487661 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:22.488373 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:22.488570 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:22.488623 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:30:22.489131 (+ 508us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:30:22.489133 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:30:22.489133 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 06:30:22.489137 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:30:22.489152 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 06:30:22.489156 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:30:22.489206 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0806 06:30:22.489260 (+ 54us) server_negotiation.cc:299] Negotiation successful
0806 06:30:22.489282 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260806 06:30:22.490767 18373 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57668:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 06:30:22.492219 18373 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 06:30:22.498185 19042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:22.494161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.129:42139 (local address 127.0.0.1:35108)
0806 06:30:22.494341 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0806 06:30:22.494376 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0806 06:30:22.494436 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:30:22.494817 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:30:22.494820 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:30:22.494823 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 06:30:22.494891 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:30:22.494893 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:30:22.496181 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:30:22.496182 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:30:22.497031 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:30:22.497033 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:30:22.497885 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:30:22.497886 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:30:22.497951 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:30:22.497962 (+ 11us) client_negotiation.cc:769] Sending connection context
0806 06:30:22.498056 (+ 94us) client_negotiation.cc:240] Negotiation successful
0806 06:30:22.498109 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"mutex_wait_us":3,"thread_start_us":64,"threads_started":1}
I20260806 06:30:22.498195 19044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:22.494247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35108 (local address 127.21.12.129:42139)
0806 06:30:22.494547 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0806 06:30:22.494550 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 06:30:22.494563 (+ 13us) server_negotiation.cc:407] Connection header received
0806 06:30:22.494604 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:30:22.494606 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:30:22.494622 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:30:22.494728 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 06:30:22.494977 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:22.496099 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:22.497107 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:22.497782 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:22.497860 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:30:22.497950 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0806 06:30:22.498075 (+ 125us) server_negotiation.cc:299] Negotiation successful
0806 06:30:22.498115 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":265,"thread_start_us":115,"threads_started":1}
I20260806 06:30:22.498389 19045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:22.494319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51240 (local address 127.21.12.131:45883)
0806 06:30:22.494470 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0806 06:30:22.494475 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 06:30:22.494683 (+ 208us) server_negotiation.cc:407] Connection header received
0806 06:30:22.494726 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:30:22.494728 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:30:22.494744 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:30:22.494821 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 06:30:22.494975 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:22.496071 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:22.497308 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:22.497954 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:22.498027 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:30:22.498104 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0806 06:30:22.498243 (+ 139us) server_negotiation.cc:299] Negotiation successful
0806 06:30:22.498308 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":71,"threads_started":1}
I20260806 06:30:22.498625 19043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:22.494163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.131:45883 (local address 127.0.0.1:51240)
0806 06:30:22.494531 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0806 06:30:22.494580 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0806 06:30:22.494628 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:30:22.494824 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:30:22.494825 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:30:22.494826 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 06:30:22.494891 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:30:22.494894 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:30:22.496154 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:30:22.496155 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:30:22.497225 (+ 1070us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:30:22.497228 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:30:22.498047 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:30:22.498048 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:30:22.498103 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:30:22.498106 (+ 3us) client_negotiation.cc:769] Sending connection context
0806 06:30:22.498164 (+ 58us) client_negotiation.cc:240] Negotiation successful
0806 06:30:22.498197 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":354,"mutex_wait_us":3,"spinlock_wait_cycles":768,"thread_start_us":80,"threads_started":1}
I20260806 06:30:22.499212 19047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:22.494376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.130:41265 (local address 127.0.0.1:51604)
0806 06:30:22.494576 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0806 06:30:22.494585 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0806 06:30:22.494626 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:30:22.495059 (+ 433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:30:22.495060 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:30:22.495061 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 06:30:22.495115 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:30:22.495116 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:30:22.495991 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:30:22.495993 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:30:22.497225 (+ 1232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:30:22.497228 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:30:22.498966 (+ 1738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:30:22.498968 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:30:22.499045 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:30:22.499049 (+ 4us) client_negotiation.cc:769] Sending connection context
0806 06:30:22.499102 (+ 53us) client_negotiation.cc:240] Negotiation successful
0806 06:30:22.499143 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":58,"threads_started":1}
I20260806 06:30:22.499230 19048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:22.494584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51604 (local address 127.21.12.130:41265)
0806 06:30:22.494783 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0806 06:30:22.494787 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 06:30:22.494796 (+ 9us) server_negotiation.cc:407] Connection header received
0806 06:30:22.494944 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:30:22.494947 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:30:22.494964 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:30:22.495189 (+ 225us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 06:30:22.495233 (+ 44us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:22.495921 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:22.498140 (+ 2219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:22.498891 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:22.498952 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:30:22.499043 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0806 06:30:22.499115 (+ 72us) server_negotiation.cc:299] Negotiation successful
0806 06:30:22.499150 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":98,"threads_started":1}
I20260806 06:30:22.499352 18543 tablet_service.cc:1467] Processing CreateTablet for tablet aa3dd5046a52437a8ea6ba19d29d2200 (DEFAULT_TABLE table=test-table [id=3b73effd720546198da98141aad4e88f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 06:30:22.499352 18936 tablet_service.cc:1467] Processing CreateTablet for tablet aa3dd5046a52437a8ea6ba19d29d2200 (DEFAULT_TABLE table=test-table [id=3b73effd720546198da98141aad4e88f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 06:30:22.499706 18936 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa3dd5046a52437a8ea6ba19d29d2200. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 06:30:22.499706 18543 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa3dd5046a52437a8ea6ba19d29d2200. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 06:30:22.502955 19049 tablet_bootstrap.cc:492] T aa3dd5046a52437a8ea6ba19d29d2200 P 68f4a0ed90d04d83a9ee1748e4a39064: Bootstrap starting.
I20260806 06:30:22.503618 19049 tablet_bootstrap.cc:654] T aa3dd5046a52437a8ea6ba19d29d2200 P 68f4a0ed90d04d83a9ee1748e4a39064: Neither blocks nor log segments found. Creating new log.
I20260806 06:30:22.503715 18741 tablet_service.cc:1467] Processing CreateTablet for tablet aa3dd5046a52437a8ea6ba19d29d2200 (DEFAULT_TABLE table=test-table [id=3b73effd720546198da98141aad4e88f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 06:30:22.503952 19049 log.cc:824] T aa3dd5046a52437a8ea6ba19d29d2200 P 68f4a0ed90d04d83a9ee1748e4a39064: Log is configured to *not* fsync() on all Append() calls
I20260806 06:30:22.504076 18741 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa3dd5046a52437a8ea6ba19d29d2200. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 06:30:22.504314 19051 tablet_bootstrap.cc:492] T aa3dd5046a52437a8ea6ba19d29d2200 P 09e36e01849540a6aa725ca9e4b0f6f9: Bootstrap starting.
I20260806 06:30:22.504644 19049 tablet_bootstrap.cc:492] T aa3dd5046a52437a8ea6ba19d29d2200 P 68f4a0ed90d04d83a9ee1748e4a39064: No bootstrap required, opened a new log
I20260806 06:30:22.504703 19049 ts_tablet_manager.cc:1397] T aa3dd5046a52437a8ea6ba19d29d2200 P 68f4a0ed90d04d83a9ee1748e4a39064: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260806 06:30:22.504931 19051 tablet_bootstrap.cc:654] T aa3dd5046a52437a8ea6ba19d29d2200 P 09e36e01849540a6aa725ca9e4b0f6f9: Neither blocks nor log segments found. Creating new log.
I20260806 06:30:22.505261 19051 log.cc:824] T aa3dd5046a52437a8ea6ba19d29d2200 P 09e36e01849540a6aa725ca9e4b0f6f9: Log is configured to *not* fsync() on all Append() calls
I20260806 06:30:22.506035 19051 tablet_bootstrap.cc:492] T aa3dd5046a52437a8ea6ba19d29d2200 P 09e36e01849540a6aa725ca9e4b0f6f9: No bootstrap required, opened a new log
I20260806 06:30:22.506107 19051 ts_tablet_manager.cc:1397] T aa3dd5046a52437a8ea6ba19d29d2200 P 09e36e01849540a6aa725ca9e4b0f6f9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260806 06:30:22.507030 19058 tablet_bootstrap.cc:492] T aa3dd5046a52437a8ea6ba19d29d2200 P ba8ce598e8bf408bad298fd0ff39238a: Bootstrap starting.
I20260806 06:30:22.507022 19049 raft_consensus.cc:348] T aa3dd5046a52437a8ea6ba19d29d2200 P 68f4a0ed90d04d83a9ee1748e4a39064 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68f4a0ed90d04d83a9ee1748e4a39064" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 45883 } } peers { permanent_uuid: "09e36e01849540a6aa725ca9e4b0f6f9" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 42139 } } peers { permanent_uuid: "ba8ce598e8bf408bad298fd0ff39238a" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 41265 } }
I20260806 06:30:22.507186 19049 raft_consensus.cc:374] T aa3dd5046a52437a8ea6ba19d29d2200 P 68f4a0ed90d04d83a9ee1748e4a39064 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 06:30:22.507227 19049 raft_consensus.cc:729] T aa3dd5046a52437a8ea6ba19d29d2200 P 68f4a0ed90d04d83a9ee1748e4a39064 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68f4a0ed90d04d83a9ee1748e4a39064, State: Initialized, Role: FOLLOWER
I20260806 06:30:22.507339 19049 consensus_queue.cc:260] T aa3dd5046a52437a8ea6ba19d29d2200 P 68f4a0ed90d04d83a9ee1748e4a39064 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68f4a0ed90d04d83a9ee1748e4a39064" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 45883 } } peers { permanent_uuid: "09e36e01849540a6aa725ca9e4b0f6f9" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 42139 } } peers { permanent_uuid: "ba8ce598e8bf408bad298fd0ff39238a" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 41265 } }
I20260806 06:30:22.507601 19049 ts_tablet_manager.cc:1428] T aa3dd5046a52437a8ea6ba19d29d2200 P 68f4a0ed90d04d83a9ee1748e4a39064: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260806 06:30:22.507742 19031 heartbeater.cc:503] Master 127.21.12.190:41463 was elected leader, sending a full tablet report...
I20260806 06:30:22.507789 19058 tablet_bootstrap.cc:654] T aa3dd5046a52437a8ea6ba19d29d2200 P ba8ce598e8bf408bad298fd0ff39238a: Neither blocks nor log segments found. Creating new log.
I20260806 06:30:22.508096 19058 log.cc:824] T aa3dd5046a52437a8ea6ba19d29d2200 P ba8ce598e8bf408bad298fd0ff39238a: Log is configured to *not* fsync() on all Append() calls
I20260806 06:30:22.508356 19051 raft_consensus.cc:348] T aa3dd5046a52437a8ea6ba19d29d2200 P 09e36e01849540a6aa725ca9e4b0f6f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68f4a0ed90d04d83a9ee1748e4a39064" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 45883 } } peers { permanent_uuid: "09e36e01849540a6aa725ca9e4b0f6f9" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 42139 } } peers { permanent_uuid: "ba8ce598e8bf408bad298fd0ff39238a" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 41265 } }
I20260806 06:30:22.508515 19051 raft_consensus.cc:374] T aa3dd5046a52437a8ea6ba19d29d2200 P 09e36e01849540a6aa725ca9e4b0f6f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 06:30:22.508563 19051 raft_consensus.cc:729] T aa3dd5046a52437a8ea6ba19d29d2200 P 09e36e01849540a6aa725ca9e4b0f6f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09e36e01849540a6aa725ca9e4b0f6f9, State: Initialized, Role: FOLLOWER
I20260806 06:30:22.508667 19051 consensus_queue.cc:260] T aa3dd5046a52437a8ea6ba19d29d2200 P 09e36e01849540a6aa725ca9e4b0f6f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68f4a0ed90d04d83a9ee1748e4a39064" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 45883 } } peers { permanent_uuid: "09e36e01849540a6aa725ca9e4b0f6f9" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 42139 } } peers { permanent_uuid: "ba8ce598e8bf408bad298fd0ff39238a" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 41265 } }
I20260806 06:30:22.508878 19058 tablet_bootstrap.cc:492] T aa3dd5046a52437a8ea6ba19d29d2200 P ba8ce598e8bf408bad298fd0ff39238a: No bootstrap required, opened a new log
I20260806 06:30:22.508924 19051 ts_tablet_manager.cc:1428] T aa3dd5046a52437a8ea6ba19d29d2200 P 09e36e01849540a6aa725ca9e4b0f6f9: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260806 06:30:22.508940 19058 ts_tablet_manager.cc:1397] T aa3dd5046a52437a8ea6ba19d29d2200 P ba8ce598e8bf408bad298fd0ff39238a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260806 06:30:22.509004 18644 heartbeater.cc:503] Master 127.21.12.190:41463 was elected leader, sending a full tablet report...
I20260806 06:30:22.510833 19058 raft_consensus.cc:348] T aa3dd5046a52437a8ea6ba19d29d2200 P ba8ce598e8bf408bad298fd0ff39238a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68f4a0ed90d04d83a9ee1748e4a39064" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 45883 } } peers { permanent_uuid: "09e36e01849540a6aa725ca9e4b0f6f9" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 42139 } } peers { permanent_uuid: "ba8ce598e8bf408bad298fd0ff39238a" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 41265 } }
I20260806 06:30:22.510947 19058 raft_consensus.cc:374] T aa3dd5046a52437a8ea6ba19d29d2200 P ba8ce598e8bf408bad298fd0ff39238a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 06:30:22.510967 19058 raft_consensus.cc:729] T aa3dd5046a52437a8ea6ba19d29d2200 P ba8ce598e8bf408bad298fd0ff39238a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba8ce598e8bf408bad298fd0ff39238a, State: Initialized, Role: FOLLOWER
I20260806 06:30:22.511065 19058 consensus_queue.cc:260] T aa3dd5046a52437a8ea6ba19d29d2200 P ba8ce598e8bf408bad298fd0ff39238a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68f4a0ed90d04d83a9ee1748e4a39064" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 45883 } } peers { permanent_uuid: "09e36e01849540a6aa725ca9e4b0f6f9" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 42139 } } peers { permanent_uuid: "ba8ce598e8bf408bad298fd0ff39238a" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 41265 } }
I20260806 06:30:22.511302 19058 ts_tablet_manager.cc:1428] T aa3dd5046a52437a8ea6ba19d29d2200 P ba8ce598e8bf408bad298fd0ff39238a: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260806 06:30:22.511337 18839 heartbeater.cc:503] Master 127.21.12.190:41463 was elected leader, sending a full tablet report...
W20260806 06:30:22.604267 18840 tablet.cc:2369] T aa3dd5046a52437a8ea6ba19d29d2200 P ba8ce598e8bf408bad298fd0ff39238a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 06:30:22.722419 18645 tablet.cc:2369] T aa3dd5046a52437a8ea6ba19d29d2200 P 09e36e01849540a6aa725ca9e4b0f6f9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 06:30:22.725682 19032 tablet.cc:2369] T aa3dd5046a52437a8ea6ba19d29d2200 P 68f4a0ed90d04d83a9ee1748e4a39064: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 06:30:22.779052 19063 raft_consensus.cc:482] T aa3dd5046a52437a8ea6ba19d29d2200 P ba8ce598e8bf408bad298fd0ff39238a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 06:30:22.779187 19063 raft_consensus.cc:504] T aa3dd5046a52437a8ea6ba19d29d2200 P ba8ce598e8bf408bad298fd0ff39238a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68f4a0ed90d04d83a9ee1748e4a39064" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 45883 } } peers { permanent_uuid: "09e36e01849540a6aa725ca9e4b0f6f9" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 42139 } } peers { permanent_uuid: "ba8ce598e8bf408bad298fd0ff39238a" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 41265 } }
I20260806 06:30:22.779510 19063 leader_election.cc:288] T aa3dd5046a52437a8ea6ba19d29d2200 P ba8ce598e8bf408bad298fd0ff39238a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 68f4a0ed90d04d83a9ee1748e4a39064 (127.21.12.131:45883), 09e36e01849540a6aa725ca9e4b0f6f9 (127.21.12.129:42139)
I20260806 06:30:22.783416 19044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:22.779782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.130:34129 (local address 127.21.12.129:42139)
0806 06:30:22.779853 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0806 06:30:22.779856 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 06:30:22.779866 (+ 10us) server_negotiation.cc:407] Connection header received
0806 06:30:22.779899 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:30:22.779900 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:30:22.779913 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:30:22.780026 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 06:30:22.780204 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:22.780988 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:22.782504 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:22.783074 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:22.783148 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:30:22.783156 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0806 06:30:22.783305 (+ 149us) server_negotiation.cc:299] Negotiation successful
0806 06:30:22.783341 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260806 06:30:22.783416 19045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:22.779910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.21.12.130:36361 (local address 127.21.12.131:45883)
0806 06:30:22.779956 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0806 06:30:22.779957 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0806 06:30:22.780015 (+ 58us) server_negotiation.cc:407] Connection header received
0806 06:30:22.780101 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:30:22.780102 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:30:22.780111 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:30:22.780160 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 06:30:22.780289 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:22.781002 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:22.782532 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:22.783068 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:22.783148 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:30:22.783156 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0806 06:30:22.783305 (+ 149us) server_negotiation.cc:299] Negotiation successful
0806 06:30:22.783344 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260806 06:30:22.783435 18842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:22.779694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.129:42139 (local address 127.21.12.130:34129)
0806 06:30:22.779775 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0806 06:30:22.779787 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 06:30:22.779827 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:30:22.780055 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:30:22.780055 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:30:22.780057 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 06:30:22.780128 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:30:22.780130 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:30:22.781068 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:30:22.781069 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:30:22.782398 (+ 1329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:30:22.782402 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:30:22.783148 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:30:22.783149 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:30:22.783235 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:30:22.783240 (+ 5us) client_negotiation.cc:769] Sending connection context
0806 06:30:22.783294 (+ 54us) client_negotiation.cc:240] Negotiation successful
0806 06:30:22.783341 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61,"mutex_wait_us":48}
I20260806 06:30:22.783440 19064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:22.779701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.131:45883 (local address 127.21.12.130:36361)
0806 06:30:22.779966 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0806 06:30:22.779975 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0806 06:30:22.780039 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:30:22.780172 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:30:22.780173 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:30:22.780175 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 06:30:22.780231 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:30:22.780232 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:30:22.781068 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:30:22.781069 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:30:22.782398 (+ 1329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:30:22.782402 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:30:22.783149 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:30:22.783150 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:30:22.783245 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:30:22.783248 (+ 3us) client_negotiation.cc:769] Sending connection context
0806 06:30:22.783294 (+ 46us) client_negotiation.cc:240] Negotiation successful
0806 06:30:22.783341 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"mutex_wait_us":175,"thread_start_us":111,"threads_started":1}
I20260806 06:30:22.783805 18573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa3dd5046a52437a8ea6ba19d29d2200" candidate_uuid: "ba8ce598e8bf408bad298fd0ff39238a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09e36e01849540a6aa725ca9e4b0f6f9" is_pre_election: true
I20260806 06:30:22.783972 18573 raft_consensus.cc:2454] T aa3dd5046a52437a8ea6ba19d29d2200 P 09e36e01849540a6aa725ca9e4b0f6f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba8ce598e8bf408bad298fd0ff39238a in term 0.
I20260806 06:30:22.784015 18967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa3dd5046a52437a8ea6ba19d29d2200" candidate_uuid: "ba8ce598e8bf408bad298fd0ff39238a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68f4a0ed90d04d83a9ee1748e4a39064" is_pre_election: true
I20260806 06:30:22.784163 18967 raft_consensus.cc:2454] T aa3dd5046a52437a8ea6ba19d29d2200 P 68f4a0ed90d04d83a9ee1748e4a39064 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba8ce598e8bf408bad298fd0ff39238a in term 0.
I20260806 06:30:22.784246 18667 leader_election.cc:302] T aa3dd5046a52437a8ea6ba19d29d2200 P ba8ce598e8bf408bad298fd0ff39238a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09e36e01849540a6aa725ca9e4b0f6f9, ba8ce598e8bf408bad298fd0ff39238a; no voters:
I20260806 06:30:22.784412 19063 raft_consensus.cc:2781] T aa3dd5046a52437a8ea6ba19d29d2200 P ba8ce598e8bf408bad298fd0ff39238a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 06:30:22.784452 19063 raft_consensus.cc:482] T aa3dd5046a52437a8ea6ba19d29d2200 P ba8ce598e8bf408bad298fd0ff39238a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 06:30:22.784474 19063 raft_consensus.cc:3037] T aa3dd5046a52437a8ea6ba19d29d2200 P ba8ce598e8bf408bad298fd0ff39238a [term 0 FOLLOWER]: Advancing to term 1
I20260806 06:30:22.785164 19063 raft_consensus.cc:504] T aa3dd5046a52437a8ea6ba19d29d2200 P ba8ce598e8bf408bad298fd0ff39238a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68f4a0ed90d04d83a9ee1748e4a39064" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 45883 } } peers { permanent_uuid: "09e36e01849540a6aa725ca9e4b0f6f9" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 42139 } } peers { permanent_uuid: "ba8ce598e8bf408bad298fd0ff39238a" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 41265 } }
I20260806 06:30:22.785300 19063 leader_election.cc:288] T aa3dd5046a52437a8ea6ba19d29d2200 P ba8ce598e8bf408bad298fd0ff39238a [CANDIDATE]: Term 1 election: Requested vote from peers 68f4a0ed90d04d83a9ee1748e4a39064 (127.21.12.131:45883), 09e36e01849540a6aa725ca9e4b0f6f9 (127.21.12.129:42139)
I20260806 06:30:22.785444 18967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa3dd5046a52437a8ea6ba19d29d2200" candidate_uuid: "ba8ce598e8bf408bad298fd0ff39238a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68f4a0ed90d04d83a9ee1748e4a39064"
I20260806 06:30:22.785446 18573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa3dd5046a52437a8ea6ba19d29d2200" candidate_uuid: "ba8ce598e8bf408bad298fd0ff39238a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09e36e01849540a6aa725ca9e4b0f6f9"
I20260806 06:30:22.785521 18573 raft_consensus.cc:3037] T aa3dd5046a52437a8ea6ba19d29d2200 P 09e36e01849540a6aa725ca9e4b0f6f9 [term 0 FOLLOWER]: Advancing to term 1
I20260806 06:30:22.785544 18967 raft_consensus.cc:3037] T aa3dd5046a52437a8ea6ba19d29d2200 P 68f4a0ed90d04d83a9ee1748e4a39064 [term 0 FOLLOWER]: Advancing to term 1
I20260806 06:30:22.786325 18573 raft_consensus.cc:2454] T aa3dd5046a52437a8ea6ba19d29d2200 P 09e36e01849540a6aa725ca9e4b0f6f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba8ce598e8bf408bad298fd0ff39238a in term 1.
I20260806 06:30:22.786341 18967 raft_consensus.cc:2454] T aa3dd5046a52437a8ea6ba19d29d2200 P 68f4a0ed90d04d83a9ee1748e4a39064 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba8ce598e8bf408bad298fd0ff39238a in term 1.
I20260806 06:30:22.786480 18667 leader_election.cc:302] T aa3dd5046a52437a8ea6ba19d29d2200 P ba8ce598e8bf408bad298fd0ff39238a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 09e36e01849540a6aa725ca9e4b0f6f9, 68f4a0ed90d04d83a9ee1748e4a39064, ba8ce598e8bf408bad298fd0ff39238a; no voters:
I20260806 06:30:22.786585 19063 raft_consensus.cc:2781] T aa3dd5046a52437a8ea6ba19d29d2200 P ba8ce598e8bf408bad298fd0ff39238a [term 1 FOLLOWER]: Leader election won for term 1
I20260806 06:30:22.786700 19063 raft_consensus.cc:686] T aa3dd5046a52437a8ea6ba19d29d2200 P ba8ce598e8bf408bad298fd0ff39238a [term 1 LEADER]: Becoming Leader. State: Replica: ba8ce598e8bf408bad298fd0ff39238a, State: Running, Role: LEADER
I20260806 06:30:22.786780 19063 consensus_queue.cc:237] T aa3dd5046a52437a8ea6ba19d29d2200 P ba8ce598e8bf408bad298fd0ff39238a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68f4a0ed90d04d83a9ee1748e4a39064" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 45883 } } peers { permanent_uuid: "09e36e01849540a6aa725ca9e4b0f6f9" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 42139 } } peers { permanent_uuid: "ba8ce598e8bf408bad298fd0ff39238a" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 41265 } }
I20260806 06:30:22.787595 18373 catalog_manager.cc:5495] T aa3dd5046a52437a8ea6ba19d29d2200 P ba8ce598e8bf408bad298fd0ff39238a reported cstate change: term changed from 0 to 1, leader changed from <none> to ba8ce598e8bf408bad298fd0ff39238a (127.21.12.130). New cstate: current_term: 1 leader_uuid: "ba8ce598e8bf408bad298fd0ff39238a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68f4a0ed90d04d83a9ee1748e4a39064" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 45883 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "09e36e01849540a6aa725ca9e4b0f6f9" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 42139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ba8ce598e8bf408bad298fd0ff39238a" member_type: VOTER last_known_addr { host: "127.21.12.130" port: 41265 } health_report { overall_health: HEALTHY } } }
I20260806 06:30:22.851022 19048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:22.848503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51620 (local address 127.21.12.130:41265)
0806 06:30:22.848566 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0806 06:30:22.848568 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:30:22.848579 (+ 11us) server_negotiation.cc:407] Connection header received
0806 06:30:22.848643 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:30:22.848644 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:30:22.848669 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:30:22.848732 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 06:30:22.848876 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:22.849399 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:22.850196 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:22.850427 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:22.850493 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:30:22.850654 (+ 161us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 06:30:22.850768 (+ 114us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 06:30:22.850850 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0806 06:30:22.850889 (+ 39us) server_negotiation.cc:299] Negotiation successful
0806 06:30:22.850909 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260806 06:30:22.852864 18967 raft_consensus.cc:1261] T aa3dd5046a52437a8ea6ba19d29d2200 P 68f4a0ed90d04d83a9ee1748e4a39064 [term 1 FOLLOWER]: Refusing update from remote peer ba8ce598e8bf408bad298fd0ff39238a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 06:30:22.852869 18573 raft_consensus.cc:1261] T aa3dd5046a52437a8ea6ba19d29d2200 P 09e36e01849540a6aa725ca9e4b0f6f9 [term 1 FOLLOWER]: Refusing update from remote peer ba8ce598e8bf408bad298fd0ff39238a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 06:30:22.853223 19065 consensus_queue.cc:1035] T aa3dd5046a52437a8ea6ba19d29d2200 P ba8ce598e8bf408bad298fd0ff39238a [LEADER]: Connected to new peer: Peer: permanent_uuid: "09e36e01849540a6aa725ca9e4b0f6f9" member_type: VOTER last_known_addr { host: "127.21.12.129" port: 42139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 06:30:22.853353 19063 consensus_queue.cc:1035] T aa3dd5046a52437a8ea6ba19d29d2200 P ba8ce598e8bf408bad298fd0ff39238a [LEADER]: Connected to new peer: Peer: permanent_uuid: "68f4a0ed90d04d83a9ee1748e4a39064" member_type: VOTER last_known_addr { host: "127.21.12.131" port: 45883 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 06:30:22.855798 19070 mvcc.cc:205] Tried to move back new op lower bound from 7315447082402500608 to 7315447082135158784. Current Snapshot: MvccSnapshot[applied={T|T < 7315447082402500608}]
I20260806 06:30:22.855911 19076 mvcc.cc:205] Tried to move back new op lower bound from 7315447082402500608 to 7315447082135158784. Current Snapshot: MvccSnapshot[applied={T|T < 7315447082402500608}]
I20260806 06:30:22.855978 19071 mvcc.cc:205] Tried to move back new op lower bound from 7315447082402500608 to 7315447082135158784. Current Snapshot: MvccSnapshot[applied={T|T < 7315447082402500608}]
I20260806 06:30:22.858508 18373 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57668:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 06:30:22.858613 18373 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57668:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 06:30:22.859295 18373 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0e835e0f83014cce8e5cd1832af65fb8: Sending DeleteTablet for 3 replicas of tablet aa3dd5046a52437a8ea6ba19d29d2200
I20260806 06:30:22.859620 18936 tablet_service.cc:1514] Processing DeleteTablet for tablet aa3dd5046a52437a8ea6ba19d29d2200 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 06:30:22 UTC) from {username='slave'} at 127.0.0.1:51240
I20260806 06:30:22.859624 18741 tablet_service.cc:1514] Processing DeleteTablet for tablet aa3dd5046a52437a8ea6ba19d29d2200 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 06:30:22 UTC) from {username='slave'} at 127.0.0.1:51604
I20260806 06:30:22.859902 18543 tablet_service.cc:1514] Processing DeleteTablet for tablet aa3dd5046a52437a8ea6ba19d29d2200 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 06:30:22 UTC) from {username='slave'} at 127.0.0.1:35108
I20260806 06:30:22.860023 21554 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHhLhT/build/release/bin/kudu with pid 18462
I20260806 06:30:22.860144 19087 tablet_replica.cc:330] T aa3dd5046a52437a8ea6ba19d29d2200 P ba8ce598e8bf408bad298fd0ff39238a: stopping tablet replica
I20260806 06:30:22.860168 19089 tablet_replica.cc:330] T aa3dd5046a52437a8ea6ba19d29d2200 P 09e36e01849540a6aa725ca9e4b0f6f9: stopping tablet replica
I20260806 06:30:22.860244 19087 raft_consensus.cc:2229] T aa3dd5046a52437a8ea6ba19d29d2200 P ba8ce598e8bf408bad298fd0ff39238a [term 1 LEADER]: Raft consensus shutting down.
I20260806 06:30:22.860363 19087 raft_consensus.cc:2258] T aa3dd5046a52437a8ea6ba19d29d2200 P ba8ce598e8bf408bad298fd0ff39238a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 06:30:22.860365 19089 raft_consensus.cc:2229] T aa3dd5046a52437a8ea6ba19d29d2200 P 09e36e01849540a6aa725ca9e4b0f6f9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 06:30:22.860795 19087 ts_tablet_manager.cc:1905] T aa3dd5046a52437a8ea6ba19d29d2200 P ba8ce598e8bf408bad298fd0ff39238a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 06:30:22.862483 19087 ts_tablet_manager.cc:1918] T aa3dd5046a52437a8ea6ba19d29d2200 P ba8ce598e8bf408bad298fd0ff39238a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 06:30:22.862557 19087 log.cc:1196] T aa3dd5046a52437a8ea6ba19d29d2200 P ba8ce598e8bf408bad298fd0ff39238a: Deleting WAL directory at /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785997581049705-21554-0/minicluster-data/ts-1/wal/wals/aa3dd5046a52437a8ea6ba19d29d2200
I20260806 06:30:22.862800 19087 ts_tablet_manager.cc:1939] T aa3dd5046a52437a8ea6ba19d29d2200 P ba8ce598e8bf408bad298fd0ff39238a: Deleting consensus metadata
I20260806 06:30:22.863237 18354 catalog_manager.cc:4841] TS ba8ce598e8bf408bad298fd0ff39238a (127.21.12.130:41265): tablet aa3dd5046a52437a8ea6ba19d29d2200 (table test-table [id=3b73effd720546198da98141aad4e88f]) successfully deleted
I20260806 06:30:22.870399 19088 tablet_replica.cc:330] T aa3dd5046a52437a8ea6ba19d29d2200 P 68f4a0ed90d04d83a9ee1748e4a39064: stopping tablet replica
I20260806 06:30:22.870540 19088 raft_consensus.cc:2229] T aa3dd5046a52437a8ea6ba19d29d2200 P 68f4a0ed90d04d83a9ee1748e4a39064 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 06:30:22.870640 19088 raft_consensus.cc:2258] T aa3dd5046a52437a8ea6ba19d29d2200 P 68f4a0ed90d04d83a9ee1748e4a39064 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 06:30:22.883179 19088 ts_tablet_manager.cc:1905] T aa3dd5046a52437a8ea6ba19d29d2200 P 68f4a0ed90d04d83a9ee1748e4a39064: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 06:30:22.884871 19088 ts_tablet_manager.cc:1918] T aa3dd5046a52437a8ea6ba19d29d2200 P 68f4a0ed90d04d83a9ee1748e4a39064: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 06:30:22.884958 19088 log.cc:1196] T aa3dd5046a52437a8ea6ba19d29d2200 P 68f4a0ed90d04d83a9ee1748e4a39064: Deleting WAL directory at /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785997581049705-21554-0/minicluster-data/ts-2/wal/wals/aa3dd5046a52437a8ea6ba19d29d2200
I20260806 06:30:22.885241 19088 ts_tablet_manager.cc:1939] T aa3dd5046a52437a8ea6ba19d29d2200 P 68f4a0ed90d04d83a9ee1748e4a39064: Deleting consensus metadata
I20260806 06:30:22.885730 18354 catalog_manager.cc:4841] TS 68f4a0ed90d04d83a9ee1748e4a39064 (127.21.12.131:45883): tablet aa3dd5046a52437a8ea6ba19d29d2200 (table test-table [id=3b73effd720546198da98141aad4e88f]) successfully deleted
W20260806 06:30:22.886909 18352 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.21.12.129:42139 (error 108)
I20260806 06:30:22.887413 19043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:22.887273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.129:42139 (local address 127.0.0.1:35110)
0806 06:30:22.887330 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0806 06:30:22.887376 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.129:42139: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260806 06:30:22.887467 21554 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHhLhT/build/release/bin/kudu with pid 18647
W20260806 06:30:22.892046 18352 catalog_manager.cc:4584] TS 09e36e01849540a6aa725ca9e4b0f6f9 (127.21.12.129:42139): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet aa3dd5046a52437a8ea6ba19d29d2200: Network error: Client connection negotiation failed: client connection to 127.21.12.129:42139: connect: Connection refused (error 111)
I20260806 06:30:22.896757 21554 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHhLhT/build/release/bin/kudu with pid 18847
I20260806 06:30:22.903285 21554 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHhLhT/build/release/bin/kudu with pid 18329
2026-08-06T06:30:22Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (1629 ms)
[----------] 6 tests from EncryptionPolicyTest (9502 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785997581049705-21554-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 06:30:22 dist-test-slave-4tdj krb5kdc[19093](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 06:30:22 dist-test-slave-4tdj krb5kdc[19093](info): set up 2 sockets
Aug 06 06:30:22 dist-test-slave-4tdj krb5kdc[19093](info): commencing operation
krb5kdc: starting...
W20260806 06:30:24.017695 21554 mini_kdc.cc:121] Time spent starting KDC: real 1.093s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785997581049705-21554-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785997581049705-21554-0/krb5kdc/test-user.keytab.
Aug 06 06:30:24 dist-test-slave-4tdj krb5kdc[19093](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997824, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T06:30:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T06:30:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785997581049705-21554-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785997581049705-21554-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 06:30:24.132449 21554 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHhLhT/build/release/bin/kudu
/tmp/dist-test-taskFHhLhT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785997581049705-21554-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785997581049705-21554-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785997581049705-21554-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785997581049705-21554-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39251
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785997581049705-21554-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46359
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39251
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785997581049705-21554-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785997581049705-21554-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785997581049705-21554-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 06:30:24.214030 19110 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 06:30:24.214246 19110 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 06:30:24.214269 19110 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 06:30:24.215433 19110 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 06:30:24.215476 19110 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 06:30:24.215488 19110 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 06:30:24.215499 19110 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 06:30:24.215513 19110 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 06:30:24.216940 19110 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46359
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785997581049705-21554-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785997581049705-21554-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39251
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785997581049705-21554-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39251
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785997581049705-21554-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19110
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785997581049705-21554-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 05:11:02 UTC on ccycloud.quasar-hvzbbz.root.comops.site
build id 4569617
I20260806 06:30:24.217134 19110 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 06:30:24.217307 19110 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 06:30:24.221968 19117 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:30:24.222095 19110 server_base.cc:1034] running on GCE node
W20260806 06:30:24.221961 19118 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:30:24.221963 19120 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:30:24.222473 19110 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 06:30:24.222784 19110 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 06:30:24.223920 19110 hybrid_clock.cc:648] HybridClock initialized: now 1785997824223907 us; error 30 us; skew 500 ppm
Aug 06 06:30:24 dist-test-slave-4tdj krb5kdc[19093](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997824, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 06:30:24.226316 19110 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 06:30:24.226742 19110 webserver.cc:460] Webserver started at http://127.0.0.1:33889/ using document root <none> and password file <none>
I20260806 06:30:24.226915 19110 fs_manager.cc:359] Metadata directory not provided
I20260806 06:30:24.226959 19110 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 06:30:24.227082 19110 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 06:30:24.228086 19110 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785997581049705-21554-0/minicluster-data/master-0/data/instance:
uuid: "653d0142396c42cd9fd4ac429d3b31ff"
format_stamp: "Formatted at 2026-08-06 06:30:24 on dist-test-slave-4tdj"
server_key: "a4e9163f4641f6413e32fa10efe7c8d3"
server_key_iv: "dc2bb6358d6f5d4e69b0dcef08ea6b60"
server_key_version: "encryptionkey@0"
I20260806 06:30:24.228406 19110 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785997581049705-21554-0/minicluster-data/master-0/wal/instance:
uuid: "653d0142396c42cd9fd4ac429d3b31ff"
format_stamp: "Formatted at 2026-08-06 06:30:24 on dist-test-slave-4tdj"
server_key: "a4e9163f4641f6413e32fa10efe7c8d3"
server_key_iv: "dc2bb6358d6f5d4e69b0dcef08ea6b60"
server_key_version: "encryptionkey@0"
I20260806 06:30:24.229656 19110 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260806 06:30:24.230325 19133 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 06:30:24.230479 19110 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260806 06:30:24.230564 19110 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785997581049705-21554-0/minicluster-data/master-0/data,/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785997581049705-21554-0/minicluster-data/master-0/wal
uuid: "653d0142396c42cd9fd4ac429d3b31ff"
format_stamp: "Formatted at 2026-08-06 06:30:24 on dist-test-slave-4tdj"
server_key: "a4e9163f4641f6413e32fa10efe7c8d3"
server_key_iv: "dc2bb6358d6f5d4e69b0dcef08ea6b60"
server_key_version: "encryptionkey@0"
I20260806 06:30:24.230645 19110 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785997581049705-21554-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785997581049705-21554-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785997581049705-21554-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 06:30:24.244519 19110 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 06:30:24.246655 19110 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 06:30:24.246791 19110 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 06:30:24.250655 19110 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39251
I20260806 06:30:24.250695 19210 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39251 every 8 connection(s)
I20260806 06:30:24.251024 19110 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785997581049705-21554-0/minicluster-data/master-0/data/info.pb
I20260806 06:30:24.251899 19212 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 06:30:24.254261 19212 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 653d0142396c42cd9fd4ac429d3b31ff: Bootstrap starting.
I20260806 06:30:24.254885 19212 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 653d0142396c42cd9fd4ac429d3b31ff: Neither blocks nor log segments found. Creating new log.
I20260806 06:30:24.255127 19212 log.cc:824] T 00000000000000000000000000000000 P 653d0142396c42cd9fd4ac429d3b31ff: Log is configured to *not* fsync() on all Append() calls
I20260806 06:30:24.255812 19212 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 653d0142396c42cd9fd4ac429d3b31ff: No bootstrap required, opened a new log
I20260806 06:30:24.257494 21554 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHhLhT/build/release/bin/kudu as pid 19110
I20260806 06:30:24.257608 21554 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785997581049705-21554-0/minicluster-data/master-0/wal/instance
I20260806 06:30:24.257490 19212 raft_consensus.cc:348] T 00000000000000000000000000000000 P 653d0142396c42cd9fd4ac429d3b31ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "653d0142396c42cd9fd4ac429d3b31ff" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39251 } }
I20260806 06:30:24.257687 19212 raft_consensus.cc:374] T 00000000000000000000000000000000 P 653d0142396c42cd9fd4ac429d3b31ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 06:30:24.257732 19212 raft_consensus.cc:729] T 00000000000000000000000000000000 P 653d0142396c42cd9fd4ac429d3b31ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 653d0142396c42cd9fd4ac429d3b31ff, State: Initialized, Role: FOLLOWER
I20260806 06:30:24.257826 21554 external_mini_cluster.cc:1428] Setting key 8ec33c156c6bdc6b1418d03ac5cde2f9
I20260806 06:30:24.257892 19212 consensus_queue.cc:260] T 00000000000000000000000000000000 P 653d0142396c42cd9fd4ac429d3b31ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "653d0142396c42cd9fd4ac429d3b31ff" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39251 } }
I20260806 06:30:24.257980 19212 raft_consensus.cc:388] T 00000000000000000000000000000000 P 653d0142396c42cd9fd4ac429d3b31ff [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 06:30:24.258010 19212 raft_consensus.cc:482] T 00000000000000000000000000000000 P 653d0142396c42cd9fd4ac429d3b31ff [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 06:30:24.258059 19212 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 653d0142396c42cd9fd4ac429d3b31ff [term 0 FOLLOWER]: Advancing to term 1
I20260806 06:30:24.258790 19212 raft_consensus.cc:504] T 00000000000000000000000000000000 P 653d0142396c42cd9fd4ac429d3b31ff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "653d0142396c42cd9fd4ac429d3b31ff" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39251 } }
I20260806 06:30:24.258896 19212 leader_election.cc:302] T 00000000000000000000000000000000 P 653d0142396c42cd9fd4ac429d3b31ff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 653d0142396c42cd9fd4ac429d3b31ff; no voters:
I20260806 06:30:24.259052 19212 leader_election.cc:288] T 00000000000000000000000000000000 P 653d0142396c42cd9fd4ac429d3b31ff [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 06:30:24.259083 19220 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 653d0142396c42cd9fd4ac429d3b31ff [term 1 FOLLOWER]: Leader election won for term 1
I20260806 06:30:24.259222 19220 raft_consensus.cc:686] T 00000000000000000000000000000000 P 653d0142396c42cd9fd4ac429d3b31ff [term 1 LEADER]: Becoming Leader. State: Replica: 653d0142396c42cd9fd4ac429d3b31ff, State: Running, Role: LEADER
I20260806 06:30:24.259322 19212 sys_catalog.cc:564] T 00000000000000000000000000000000 P 653d0142396c42cd9fd4ac429d3b31ff [sys.catalog]: configured and running, proceeding with master startup.
I20260806 06:30:24.259318 19220 consensus_queue.cc:237] T 00000000000000000000000000000000 P 653d0142396c42cd9fd4ac429d3b31ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "653d0142396c42cd9fd4ac429d3b31ff" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39251 } }
I20260806 06:30:24.260566 19223 sys_catalog.cc:455] T 00000000000000000000000000000000 P 653d0142396c42cd9fd4ac429d3b31ff [sys.catalog]: SysCatalogTable state changed. Reason: New leader 653d0142396c42cd9fd4ac429d3b31ff. Latest consensus state: current_term: 1 leader_uuid: "653d0142396c42cd9fd4ac429d3b31ff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "653d0142396c42cd9fd4ac429d3b31ff" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39251 } } }
I20260806 06:30:24.260680 19223 sys_catalog.cc:458] T 00000000000000000000000000000000 P 653d0142396c42cd9fd4ac429d3b31ff [sys.catalog]: This master's current role is: LEADER
I20260806 06:30:24.260929 19222 sys_catalog.cc:455] T 00000000000000000000000000000000 P 653d0142396c42cd9fd4ac429d3b31ff [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "653d0142396c42cd9fd4ac429d3b31ff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "653d0142396c42cd9fd4ac429d3b31ff" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39251 } } }
I20260806 06:30:24.261003 19222 sys_catalog.cc:458] T 00000000000000000000000000000000 P 653d0142396c42cd9fd4ac429d3b31ff [sys.catalog]: This master's current role is: LEADER
W20260806 06:30:24.261242 19238 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 653d0142396c42cd9fd4ac429d3b31ff: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 06:30:24.261299 19238 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 06:30:24.261385 19239 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 06:30:24.261576 19239 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Aug 06 06:30:24 dist-test-slave-4tdj krb5kdc[19093](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997824, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260806 06:30:24.265931 19239 catalog_manager.cc:1347] Generated new cluster ID: f84d415652034d5bb8e2c5a20a88e81b
I20260806 06:30:24.265986 19239 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 06:30:24.270365 19219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:24.258427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60915 (local address 127.0.0.1:39251)
0806 06:30:24.258933 (+ 506us) server_negotiation.cc:206] Beginning negotiation
0806 06:30:24.258944 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 06:30:24.258980 (+ 36us) server_negotiation.cc:407] Connection header received
0806 06:30:24.259111 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:30:24.259120 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:30:24.259184 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:30:24.260264 (+ 1080us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:30:24.260410 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:24.261847 (+ 1437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:24.262894 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:24.263306 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:24.263392 (+ 86us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:30:24.265987 (+ 2595us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:30:24.265991 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:30:24.265992 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 06:30:24.266004 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:30:24.268631 (+ 2627us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:30:24.269091 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:30:24.269094 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:30:24.269095 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:30:24.269213 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:30:24.269437 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:30:24.269437 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:30:24.269438 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:30:24.269725 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:30:24.269878 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0806 06:30:24.270009 (+ 131us) server_negotiation.cc:299] Negotiation successful
0806 06:30:24.270111 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":452,"thread_start_us":404,"threads_started":1}
I20260806 06:30:24.271440 21554 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 06:30:24.276285 19239 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 06:30:24.276947 19239 catalog_manager.cc:1504] Loading token signing keys...
I20260806 06:30:24.279798 19239 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 653d0142396c42cd9fd4ac429d3b31ff: Generated new TSK 0
I20260806 06:30:24.279903 19219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:24.272864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53259 (local address 127.0.0.1:39251)
0806 06:30:24.272899 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0806 06:30:24.272901 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:30:24.273003 (+ 102us) server_negotiation.cc:407] Connection header received
0806 06:30:24.273054 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:30:24.273055 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:30:24.273076 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:30:24.273129 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:30:24.273723 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:24.274468 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:24.275455 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:24.275734 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:24.276065 (+ 331us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:30:24.277154 (+ 1089us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:30:24.277155 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:30:24.277156 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 06:30:24.277162 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:30:24.278580 (+ 1418us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:30:24.278959 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:30:24.278960 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:30:24.278960 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:30:24.279006 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:30:24.279209 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:30:24.279209 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:30:24.279210 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:30:24.279313 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:30:24.279723 (+ 410us) server_negotiation.cc:1035] Waiting for connection context
0806 06:30:24.279775 (+ 52us) server_negotiation.cc:299] Negotiation successful
0806 06:30:24.279835 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260806 06:30:24.279975 19239 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 06:30:24.281684 21554 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHhLhT/build/release/bin/kudu with pid 19110
W20260806 06:30:24.292747 19091 connection.cc:535] client connection to 127.0.0.1:39251 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39251): Connection reset by peer (error 104)
2026-08-06T06:30:24Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1381 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785997581049705-21554-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 06:30:24 dist-test-slave-4tdj krb5kdc[19251](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 06:30:24 dist-test-slave-4tdj krb5kdc[19251](info): set up 2 sockets
Aug 06 06:30:24 dist-test-slave-4tdj krb5kdc[19251](info): commencing operation
krb5kdc: starting...
W20260806 06:30:25.400295 21554 mini_kdc.cc:121] Time spent starting KDC: real 1.094s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785997581049705-21554-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785997581049705-21554-0/krb5kdc/test-user.keytab.
Aug 06 06:30:25 dist-test-slave-4tdj krb5kdc[19251](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997825, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T06:30:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T06:30:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785997581049705-21554-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785997581049705-21554-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 06:30:25.513262 21554 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHhLhT/build/release/bin/kudu
/tmp/dist-test-taskFHhLhT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785997581049705-21554-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785997581049705-21554-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785997581049705-21554-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785997581049705-21554-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40357
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785997581049705-21554-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40357
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785997581049705-21554-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785997581049705-21554-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785997581049705-21554-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 06:30:25.592633 19270 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 06:30:25.592833 19270 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 06:30:25.592864 19270 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 06:30:25.593981 19270 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 06:30:25.594022 19270 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 06:30:25.594040 19270 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 06:30:25.594058 19270 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 06:30:25.594072 19270 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 06:30:25.595553 19270 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785997581049705-21554-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785997581049705-21554-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40357
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785997581049705-21554-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40357
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785997581049705-21554-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19270
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785997581049705-21554-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 05:11:02 UTC on ccycloud.quasar-hvzbbz.root.comops.site
build id 4569617
I20260806 06:30:25.595788 19270 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 06:30:25.595985 19270 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 06:30:25.600612 19276 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:30:25.600592 19278 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:30:25.600582 19280 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:30:25.600817 19270 server_base.cc:1034] running on GCE node
I20260806 06:30:25.601040 19270 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 06:30:25.601313 19270 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 06:30:25.602460 19270 hybrid_clock.cc:648] HybridClock initialized: now 1785997825602441 us; error 33 us; skew 500 ppm
Aug 06 06:30:25 dist-test-slave-4tdj krb5kdc[19251](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997825, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 06:30:25.604729 19270 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 06:30:25.605123 19270 webserver.cc:460] Webserver started at http://127.0.0.1:44635/ using document root <none> and password file <none>
I20260806 06:30:25.605329 19270 fs_manager.cc:359] Metadata directory not provided
I20260806 06:30:25.605376 19270 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 06:30:25.605496 19270 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 06:30:25.606552 19270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785997581049705-21554-0/minicluster-data/master-0/data/instance:
uuid: "c464f7c1de7c411e816592ed4218604a"
format_stamp: "Formatted at 2026-08-06 06:30:25 on dist-test-slave-4tdj"
server_key: "526cf129d0681b7ca0303ad79ba1c767"
server_key_iv: "4b1d3b5757581b7c7ef9d260cdaf609e"
server_key_version: "encryptionkey@0"
I20260806 06:30:25.606882 19270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785997581049705-21554-0/minicluster-data/master-0/wal/instance:
uuid: "c464f7c1de7c411e816592ed4218604a"
format_stamp: "Formatted at 2026-08-06 06:30:25 on dist-test-slave-4tdj"
server_key: "526cf129d0681b7ca0303ad79ba1c767"
server_key_iv: "4b1d3b5757581b7c7ef9d260cdaf609e"
server_key_version: "encryptionkey@0"
I20260806 06:30:25.608085 19270 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260806 06:30:25.608757 19290 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 06:30:25.608906 19270 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260806 06:30:25.608989 19270 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785997581049705-21554-0/minicluster-data/master-0/data,/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785997581049705-21554-0/minicluster-data/master-0/wal
uuid: "c464f7c1de7c411e816592ed4218604a"
format_stamp: "Formatted at 2026-08-06 06:30:25 on dist-test-slave-4tdj"
server_key: "526cf129d0681b7ca0303ad79ba1c767"
server_key_iv: "4b1d3b5757581b7c7ef9d260cdaf609e"
server_key_version: "encryptionkey@0"
I20260806 06:30:25.609047 19270 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785997581049705-21554-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785997581049705-21554-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785997581049705-21554-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 06:30:25.621294 19270 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 06:30:25.623077 19270 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 06:30:25.623209 19270 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 06:30:25.626930 19270 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40357
I20260806 06:30:25.626963 19362 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40357 every 8 connection(s)
I20260806 06:30:25.627261 19270 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785997581049705-21554-0/minicluster-data/master-0/data/info.pb
I20260806 06:30:25.628094 19363 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 06:30:25.630329 19363 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c464f7c1de7c411e816592ed4218604a: Bootstrap starting.
I20260806 06:30:25.630862 19363 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c464f7c1de7c411e816592ed4218604a: Neither blocks nor log segments found. Creating new log.
I20260806 06:30:25.631093 19363 log.cc:824] T 00000000000000000000000000000000 P c464f7c1de7c411e816592ed4218604a: Log is configured to *not* fsync() on all Append() calls
I20260806 06:30:25.631703 19363 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c464f7c1de7c411e816592ed4218604a: No bootstrap required, opened a new log
I20260806 06:30:25.633211 19363 raft_consensus.cc:348] T 00000000000000000000000000000000 P c464f7c1de7c411e816592ed4218604a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c464f7c1de7c411e816592ed4218604a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40357 } }
I20260806 06:30:25.633342 19363 raft_consensus.cc:374] T 00000000000000000000000000000000 P c464f7c1de7c411e816592ed4218604a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 06:30:25.633371 19363 raft_consensus.cc:729] T 00000000000000000000000000000000 P c464f7c1de7c411e816592ed4218604a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c464f7c1de7c411e816592ed4218604a, State: Initialized, Role: FOLLOWER
I20260806 06:30:25.633466 19363 consensus_queue.cc:260] T 00000000000000000000000000000000 P c464f7c1de7c411e816592ed4218604a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c464f7c1de7c411e816592ed4218604a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40357 } }
I20260806 06:30:25.633513 19363 raft_consensus.cc:388] T 00000000000000000000000000000000 P c464f7c1de7c411e816592ed4218604a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 06:30:25.633533 19363 raft_consensus.cc:482] T 00000000000000000000000000000000 P c464f7c1de7c411e816592ed4218604a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 06:30:25.633562 19363 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c464f7c1de7c411e816592ed4218604a [term 0 FOLLOWER]: Advancing to term 1
I20260806 06:30:25.634007 19363 raft_consensus.cc:504] T 00000000000000000000000000000000 P c464f7c1de7c411e816592ed4218604a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c464f7c1de7c411e816592ed4218604a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40357 } }
I20260806 06:30:25.634101 19363 leader_election.cc:302] T 00000000000000000000000000000000 P c464f7c1de7c411e816592ed4218604a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c464f7c1de7c411e816592ed4218604a; no voters:
I20260806 06:30:25.634217 19363 leader_election.cc:288] T 00000000000000000000000000000000 P c464f7c1de7c411e816592ed4218604a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 06:30:25.634262 19368 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c464f7c1de7c411e816592ed4218604a [term 1 FOLLOWER]: Leader election won for term 1
I20260806 06:30:25.634433 19368 raft_consensus.cc:686] T 00000000000000000000000000000000 P c464f7c1de7c411e816592ed4218604a [term 1 LEADER]: Becoming Leader. State: Replica: c464f7c1de7c411e816592ed4218604a, State: Running, Role: LEADER
I20260806 06:30:25.634456 19363 sys_catalog.cc:564] T 00000000000000000000000000000000 P c464f7c1de7c411e816592ed4218604a [sys.catalog]: configured and running, proceeding with master startup.
I20260806 06:30:25.634518 19368 consensus_queue.cc:237] T 00000000000000000000000000000000 P c464f7c1de7c411e816592ed4218604a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c464f7c1de7c411e816592ed4218604a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40357 } }
I20260806 06:30:25.634900 19370 sys_catalog.cc:455] T 00000000000000000000000000000000 P c464f7c1de7c411e816592ed4218604a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c464f7c1de7c411e816592ed4218604a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c464f7c1de7c411e816592ed4218604a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40357 } } }
I20260806 06:30:25.634984 19370 sys_catalog.cc:458] T 00000000000000000000000000000000 P c464f7c1de7c411e816592ed4218604a [sys.catalog]: This master's current role is: LEADER
I20260806 06:30:25.635109 19371 sys_catalog.cc:455] T 00000000000000000000000000000000 P c464f7c1de7c411e816592ed4218604a [sys.catalog]: SysCatalogTable state changed. Reason: New leader c464f7c1de7c411e816592ed4218604a. Latest consensus state: current_term: 1 leader_uuid: "c464f7c1de7c411e816592ed4218604a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c464f7c1de7c411e816592ed4218604a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40357 } } }
I20260806 06:30:25.635221 19371 sys_catalog.cc:458] T 00000000000000000000000000000000 P c464f7c1de7c411e816592ed4218604a [sys.catalog]: This master's current role is: LEADER
I20260806 06:30:25.635402 19380 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 06:30:25.635922 19380 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 06:30:25.637485 21554 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHhLhT/build/release/bin/kudu as pid 19270
I20260806 06:30:25.637578 21554 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785997581049705-21554-0/minicluster-data/master-0/wal/instance
I20260806 06:30:25.637751 21554 external_mini_cluster.cc:1428] Setting key 7846db03fa4231568a1a10fdb18bed4d
I20260806 06:30:25.637920 19380 catalog_manager.cc:1347] Generated new cluster ID: 3cd40777551b4828be23cc2b379b480b
I20260806 06:30:25.637960 19380 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 06:30:25.647387 19395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:25.638146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36985 (local address 127.0.0.1:40357)
0806 06:30:25.642403 (+ 4257us) server_negotiation.cc:206] Beginning negotiation
0806 06:30:25.642412 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 06:30:25.642443 (+ 31us) server_negotiation.cc:407] Connection header received
0806 06:30:25.642547 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:30:25.642553 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:30:25.642618 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:30:25.642734 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:30:25.642977 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:25.644289 (+ 1312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:25.645387 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:25.645968 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:25.646056 (+ 88us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:30:25.646724 (+ 668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:30:25.646728 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:30:25.646728 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 06:30:25.646926 (+ 198us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:30:25.646961 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 06:30:25.646975 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:30:25.647051 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0806 06:30:25.647145 (+ 94us) server_negotiation.cc:299] Negotiation successful
0806 06:30:25.647170 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4206,"thread_start_us":70,"threads_started":1}
I20260806 06:30:25.648269 21554 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 06:30:25.648293 19380 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 06:30:25.648886 19380 catalog_manager.cc:1504] Loading token signing keys...
I20260806 06:30:25.653607 19395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:25.649669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51617 (local address 127.0.0.1:40357)
0806 06:30:25.649699 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0806 06:30:25.649700 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0806 06:30:25.650434 (+ 734us) server_negotiation.cc:407] Connection header received
0806 06:30:25.650481 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:30:25.650481 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:30:25.650493 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:30:25.650564 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:30:25.650669 (+ 105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:25.651437 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:25.652406 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:25.652737 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:25.652803 (+ 66us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:30:25.653378 (+ 575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:30:25.653380 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:30:25.653380 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 06:30:25.653384 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:30:25.653398 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 06:30:25.653403 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:30:25.653461 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0806 06:30:25.653513 (+ 52us) server_negotiation.cc:299] Negotiation successful
0806 06:30:25.653533 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":7,"server-negotiator.queue_time_us":15}
I20260806 06:30:25.661429 19380 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c464f7c1de7c411e816592ed4218604a: Generated new TSK 0
I20260806 06:30:25.661581 19380 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 06:30:25.664508 21554 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHhLhT/build/release/bin/kudu with pid 19270
W20260806 06:30:25.677430 19249 connection.cc:535] client connection to 127.0.0.1:40357 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40357): Connection reset by peer (error 104)
2026-08-06T06:30:25Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1384 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785997581049705-21554-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 06:30:25 dist-test-slave-4tdj krb5kdc[19404](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 06:30:25 dist-test-slave-4tdj krb5kdc[19404](info): set up 2 sockets
Aug 06 06:30:25 dist-test-slave-4tdj krb5kdc[19404](info): commencing operation
krb5kdc: starting...
W20260806 06:30:26.785034 21554 mini_kdc.cc:121] Time spent starting KDC: real 1.094s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785997581049705-21554-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785997581049705-21554-0/krb5kdc/test-user.keytab.
Aug 06 06:30:26 dist-test-slave-4tdj krb5kdc[19404](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997826, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T06:30:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T06:30:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785997581049705-21554-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785997581049705-21554-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 06:30:26.901221 21554 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHhLhT/build/release/bin/kudu
/tmp/dist-test-taskFHhLhT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785997581049705-21554-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785997581049705-21554-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785997581049705-21554-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785997581049705-21554-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36379
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785997581049705-21554-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36379
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785997581049705-21554-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785997581049705-21554-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785997581049705-21554-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 06:30:26.980465 19422 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 06:30:26.980676 19422 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 06:30:26.980708 19422 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 06:30:26.981838 19422 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 06:30:26.981879 19422 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 06:30:26.981899 19422 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 06:30:26.981914 19422 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 06:30:26.981930 19422 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 06:30:26.983418 19422 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785997581049705-21554-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785997581049705-21554-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36379
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785997581049705-21554-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36379
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785997581049705-21554-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19422
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785997581049705-21554-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 05:11:02 UTC on ccycloud.quasar-hvzbbz.root.comops.site
build id 4569617
I20260806 06:30:26.983662 19422 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 06:30:26.983865 19422 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 06:30:26.988497 19435 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:30:26.988543 19431 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:30:26.988582 19432 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:30:26.988801 19422 server_base.cc:1034] running on GCE node
I20260806 06:30:26.989015 19422 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 06:30:26.989279 19422 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 06:30:26.990412 19422 hybrid_clock.cc:648] HybridClock initialized: now 1785997826990397 us; error 29 us; skew 500 ppm
Aug 06 06:30:26 dist-test-slave-4tdj krb5kdc[19404](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997826, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 06:30:26.992867 19422 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 06:30:26.993300 19422 webserver.cc:460] Webserver started at http://127.0.0.1:44561/ using document root <none> and password file <none>
I20260806 06:30:26.993497 19422 fs_manager.cc:359] Metadata directory not provided
I20260806 06:30:26.993541 19422 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 06:30:26.993645 19422 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 06:30:26.994720 19422 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785997581049705-21554-0/minicluster-data/master-0/data/instance:
uuid: "0b6c0ecf263b46f5814274e58445bcf9"
format_stamp: "Formatted at 2026-08-06 06:30:26 on dist-test-slave-4tdj"
server_key: "369b3f3f3245f9cef0ff5b37f26b6fd7"
server_key_iv: "15b98d3a68b5f58bdaed1a63dc80f240"
server_key_version: "encryptionkey@0"
I20260806 06:30:26.995041 19422 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785997581049705-21554-0/minicluster-data/master-0/wal/instance:
uuid: "0b6c0ecf263b46f5814274e58445bcf9"
format_stamp: "Formatted at 2026-08-06 06:30:26 on dist-test-slave-4tdj"
server_key: "369b3f3f3245f9cef0ff5b37f26b6fd7"
server_key_iv: "15b98d3a68b5f58bdaed1a63dc80f240"
server_key_version: "encryptionkey@0"
I20260806 06:30:26.996253 19422 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260806 06:30:26.997081 19444 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 06:30:26.997272 19422 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 06:30:26.997359 19422 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785997581049705-21554-0/minicluster-data/master-0/data,/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785997581049705-21554-0/minicluster-data/master-0/wal
uuid: "0b6c0ecf263b46f5814274e58445bcf9"
format_stamp: "Formatted at 2026-08-06 06:30:26 on dist-test-slave-4tdj"
server_key: "369b3f3f3245f9cef0ff5b37f26b6fd7"
server_key_iv: "15b98d3a68b5f58bdaed1a63dc80f240"
server_key_version: "encryptionkey@0"
I20260806 06:30:26.997422 19422 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785997581049705-21554-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785997581049705-21554-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785997581049705-21554-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 06:30:27.000690 19422 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 06:30:27.002535 19422 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 06:30:27.002655 19422 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 06:30:27.006429 19422 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36379
I20260806 06:30:27.006464 19518 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36379 every 8 connection(s)
I20260806 06:30:27.006769 19422 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785997581049705-21554-0/minicluster-data/master-0/data/info.pb
I20260806 06:30:27.007694 19519 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 06:30:27.009905 19519 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b6c0ecf263b46f5814274e58445bcf9: Bootstrap starting.
I20260806 06:30:27.010505 19519 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0b6c0ecf263b46f5814274e58445bcf9: Neither blocks nor log segments found. Creating new log.
I20260806 06:30:27.010766 19519 log.cc:824] T 00000000000000000000000000000000 P 0b6c0ecf263b46f5814274e58445bcf9: Log is configured to *not* fsync() on all Append() calls
I20260806 06:30:27.011397 19519 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b6c0ecf263b46f5814274e58445bcf9: No bootstrap required, opened a new log
I20260806 06:30:27.012928 19519 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0b6c0ecf263b46f5814274e58445bcf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b6c0ecf263b46f5814274e58445bcf9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36379 } }
I20260806 06:30:27.013063 19519 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0b6c0ecf263b46f5814274e58445bcf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 06:30:27.013120 19519 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0b6c0ecf263b46f5814274e58445bcf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6c0ecf263b46f5814274e58445bcf9, State: Initialized, Role: FOLLOWER
I20260806 06:30:27.013254 19519 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0b6c0ecf263b46f5814274e58445bcf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b6c0ecf263b46f5814274e58445bcf9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36379 } }
I20260806 06:30:27.013317 19519 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0b6c0ecf263b46f5814274e58445bcf9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 06:30:27.013355 19519 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0b6c0ecf263b46f5814274e58445bcf9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 06:30:27.013401 19519 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0b6c0ecf263b46f5814274e58445bcf9 [term 0 FOLLOWER]: Advancing to term 1
I20260806 06:30:27.013903 19519 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0b6c0ecf263b46f5814274e58445bcf9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b6c0ecf263b46f5814274e58445bcf9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36379 } }
I20260806 06:30:27.014002 19519 leader_election.cc:302] T 00000000000000000000000000000000 P 0b6c0ecf263b46f5814274e58445bcf9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0b6c0ecf263b46f5814274e58445bcf9; no voters:
I20260806 06:30:27.014164 19519 leader_election.cc:288] T 00000000000000000000000000000000 P 0b6c0ecf263b46f5814274e58445bcf9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 06:30:27.014217 19523 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0b6c0ecf263b46f5814274e58445bcf9 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 06:30:27.014386 19523 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0b6c0ecf263b46f5814274e58445bcf9 [term 1 LEADER]: Becoming Leader. State: Replica: 0b6c0ecf263b46f5814274e58445bcf9, State: Running, Role: LEADER
I20260806 06:30:27.014436 19519 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0b6c0ecf263b46f5814274e58445bcf9 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 06:30:27.014477 19523 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0b6c0ecf263b46f5814274e58445bcf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b6c0ecf263b46f5814274e58445bcf9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36379 } }
I20260806 06:30:27.014815 19525 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b6c0ecf263b46f5814274e58445bcf9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0b6c0ecf263b46f5814274e58445bcf9. Latest consensus state: current_term: 1 leader_uuid: "0b6c0ecf263b46f5814274e58445bcf9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b6c0ecf263b46f5814274e58445bcf9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36379 } } }
I20260806 06:30:27.014896 19525 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b6c0ecf263b46f5814274e58445bcf9 [sys.catalog]: This master's current role is: LEADER
I20260806 06:30:27.015017 19524 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b6c0ecf263b46f5814274e58445bcf9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0b6c0ecf263b46f5814274e58445bcf9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b6c0ecf263b46f5814274e58445bcf9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36379 } } }
I20260806 06:30:27.015123 19524 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b6c0ecf263b46f5814274e58445bcf9 [sys.catalog]: This master's current role is: LEADER
I20260806 06:30:27.015437 19533 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 06:30:27.015568 21554 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHhLhT/build/release/bin/kudu as pid 19422
I20260806 06:30:27.015699 21554 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785997581049705-21554-0/minicluster-data/master-0/wal/instance
I20260806 06:30:27.015902 21554 external_mini_cluster.cc:1428] Setting key 1cb11515186fd3e4dad5711dd84145fd
I20260806 06:30:27.016014 19533 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 06:30:27.017613 19533 catalog_manager.cc:1347] Generated new cluster ID: e90741a4ebd642db973ef06820907e81
I20260806 06:30:27.017668 19533 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 06:30:27.018229 19545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:27.016540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44945 (local address 127.0.0.1:36379)
0806 06:30:27.016718 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0806 06:30:27.016727 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 06:30:27.016759 (+ 32us) server_negotiation.cc:407] Connection header received
0806 06:30:27.016850 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:30:27.016856 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:30:27.016879 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:30:27.016994 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:30:27.017776 (+ 782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:30:27.017780 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:30:27.017780 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 06:30:27.017785 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:30:27.017826 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 06:30:27.017841 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:30:27.017922 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0806 06:30:27.017998 (+ 76us) server_negotiation.cc:299] Negotiation successful
0806 06:30:27.018041 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":75,"threads_started":1}
I20260806 06:30:27.019225 21554 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 06:30:27.030900 19533 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 06:30:27.031093 19533 catalog_manager.cc:1504] Loading token signing keys...
I20260806 06:30:27.032670 19545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:27.027113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52569 (local address 127.0.0.1:36379)
0806 06:30:27.030540 (+ 3427us) server_negotiation.cc:206] Beginning negotiation
0806 06:30:27.030543 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 06:30:27.031310 (+ 767us) server_negotiation.cc:407] Connection header received
0806 06:30:27.031422 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:30:27.031423 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:30:27.031439 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:30:27.031518 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:30:27.032251 (+ 733us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:30:27.032253 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:30:27.032253 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 06:30:27.032257 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:30:27.032271 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 06:30:27.032276 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:30:27.032331 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0806 06:30:27.032573 (+ 242us) server_negotiation.cc:299] Negotiation successful
0806 06:30:27.032587 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3406}
I20260806 06:30:27.033869 19533 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0b6c0ecf263b46f5814274e58445bcf9: Generated new TSK 0
I20260806 06:30:27.034035 19533 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 06:30:27.039623 21554 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHhLhT/build/release/bin/kudu with pid 19422
2026-08-06T06:30:27Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1376 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785997581049705-21554-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 06:30:27 dist-test-slave-4tdj krb5kdc[19555](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 06:30:27 dist-test-slave-4tdj krb5kdc[19555](info): set up 2 sockets
Aug 06 06:30:27 dist-test-slave-4tdj krb5kdc[19555](info): commencing operation
krb5kdc: starting...
W20260806 06:30:28.170135 21554 mini_kdc.cc:121] Time spent starting KDC: real 1.103s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785997581049705-21554-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785997581049705-21554-0/krb5kdc/test-user.keytab.
Aug 06 06:30:28 dist-test-slave-4tdj krb5kdc[19555](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997828, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T06:30:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T06:30:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785997581049705-21554-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785997581049705-21554-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 06:30:28.289371 21554 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHhLhT/build/release/bin/kudu
/tmp/dist-test-taskFHhLhT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785997581049705-21554-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785997581049705-21554-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785997581049705-21554-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785997581049705-21554-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43899
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785997581049705-21554-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43899
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785997581049705-21554-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785997581049705-21554-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785997581049705-21554-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 06:30:28.370236 19576 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 06:30:28.370502 19576 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 06:30:28.370530 19576 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 06:30:28.371712 19576 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 06:30:28.371774 19576 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 06:30:28.371788 19576 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 06:30:28.371800 19576 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 06:30:28.371810 19576 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 06:30:28.373271 19576 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785997581049705-21554-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785997581049705-21554-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43899
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785997581049705-21554-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43899
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785997581049705-21554-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19576
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785997581049705-21554-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 05:11:02 UTC on ccycloud.quasar-hvzbbz.root.comops.site
build id 4569617
I20260806 06:30:28.373461 19576 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 06:30:28.373629 19576 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 06:30:28.378360 19584 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:30:28.378407 19581 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:30:28.378364 19582 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:30:28.378566 19576 server_base.cc:1034] running on GCE node
I20260806 06:30:28.378926 19576 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 06:30:28.379207 19576 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 06:30:28.380362 19576 hybrid_clock.cc:648] HybridClock initialized: now 1785997828380342 us; error 34 us; skew 500 ppm
Aug 06 06:30:28 dist-test-slave-4tdj krb5kdc[19555](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997828, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 06:30:28.382800 19576 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 06:30:28.383250 19576 webserver.cc:460] Webserver started at http://127.0.0.1:38051/ using document root <none> and password file <none>
I20260806 06:30:28.383450 19576 fs_manager.cc:359] Metadata directory not provided
I20260806 06:30:28.383496 19576 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 06:30:28.383618 19576 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 06:30:28.384701 19576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785997581049705-21554-0/minicluster-data/master-0/data/instance:
uuid: "19e8a70ac22749b9bb6a392b989d1806"
format_stamp: "Formatted at 2026-08-06 06:30:28 on dist-test-slave-4tdj"
server_key: "2ff41a4de1ffa79f7bb7444d64579d48"
server_key_iv: "c850a71592dc5458c8a0a91e34f9b063"
server_key_version: "encryptionkey@0"
I20260806 06:30:28.385032 19576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785997581049705-21554-0/minicluster-data/master-0/wal/instance:
uuid: "19e8a70ac22749b9bb6a392b989d1806"
format_stamp: "Formatted at 2026-08-06 06:30:28 on dist-test-slave-4tdj"
server_key: "2ff41a4de1ffa79f7bb7444d64579d48"
server_key_iv: "c850a71592dc5458c8a0a91e34f9b063"
server_key_version: "encryptionkey@0"
I20260806 06:30:28.386271 19576 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260806 06:30:28.387037 19595 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 06:30:28.387225 19576 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260806 06:30:28.387323 19576 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785997581049705-21554-0/minicluster-data/master-0/data,/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785997581049705-21554-0/minicluster-data/master-0/wal
uuid: "19e8a70ac22749b9bb6a392b989d1806"
format_stamp: "Formatted at 2026-08-06 06:30:28 on dist-test-slave-4tdj"
server_key: "2ff41a4de1ffa79f7bb7444d64579d48"
server_key_iv: "c850a71592dc5458c8a0a91e34f9b063"
server_key_version: "encryptionkey@0"
I20260806 06:30:28.387396 19576 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785997581049705-21554-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785997581049705-21554-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785997581049705-21554-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 06:30:28.406697 19576 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 06:30:28.408663 19576 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 06:30:28.408803 19576 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 06:30:28.412672 19576 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43899
I20260806 06:30:28.412722 19669 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43899 every 8 connection(s)
I20260806 06:30:28.413056 19576 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785997581049705-21554-0/minicluster-data/master-0/data/info.pb
I20260806 06:30:28.413911 19671 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 06:30:28.414525 21554 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHhLhT/build/release/bin/kudu as pid 19576
I20260806 06:30:28.414625 21554 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785997581049705-21554-0/minicluster-data/master-0/wal/instance
I20260806 06:30:28.414790 21554 external_mini_cluster.cc:1428] Setting key 05de3067cbd58db5519d6e674e7db762
I20260806 06:30:28.416792 19671 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 19e8a70ac22749b9bb6a392b989d1806: Bootstrap starting.
I20260806 06:30:28.417371 19671 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 19e8a70ac22749b9bb6a392b989d1806: Neither blocks nor log segments found. Creating new log.
I20260806 06:30:28.417649 19671 log.cc:824] T 00000000000000000000000000000000 P 19e8a70ac22749b9bb6a392b989d1806: Log is configured to *not* fsync() on all Append() calls
I20260806 06:30:28.418355 19671 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 19e8a70ac22749b9bb6a392b989d1806: No bootstrap required, opened a new log
I20260806 06:30:28.420641 19671 raft_consensus.cc:348] T 00000000000000000000000000000000 P 19e8a70ac22749b9bb6a392b989d1806 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19e8a70ac22749b9bb6a392b989d1806" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43899 } }
I20260806 06:30:28.420800 19671 raft_consensus.cc:374] T 00000000000000000000000000000000 P 19e8a70ac22749b9bb6a392b989d1806 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 06:30:28.420825 19671 raft_consensus.cc:729] T 00000000000000000000000000000000 P 19e8a70ac22749b9bb6a392b989d1806 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19e8a70ac22749b9bb6a392b989d1806, State: Initialized, Role: FOLLOWER
I20260806 06:30:28.420935 19671 consensus_queue.cc:260] T 00000000000000000000000000000000 P 19e8a70ac22749b9bb6a392b989d1806 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19e8a70ac22749b9bb6a392b989d1806" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43899 } }
I20260806 06:30:28.421000 19671 raft_consensus.cc:388] T 00000000000000000000000000000000 P 19e8a70ac22749b9bb6a392b989d1806 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 06:30:28.421020 19671 raft_consensus.cc:482] T 00000000000000000000000000000000 P 19e8a70ac22749b9bb6a392b989d1806 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 06:30:28.421051 19671 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 19e8a70ac22749b9bb6a392b989d1806 [term 0 FOLLOWER]: Advancing to term 1
Aug 06 06:30:28 dist-test-slave-4tdj krb5kdc[19555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997828, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260806 06:30:28.421581 19671 raft_consensus.cc:504] T 00000000000000000000000000000000 P 19e8a70ac22749b9bb6a392b989d1806 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19e8a70ac22749b9bb6a392b989d1806" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43899 } }
I20260806 06:30:28.421691 19671 leader_election.cc:302] T 00000000000000000000000000000000 P 19e8a70ac22749b9bb6a392b989d1806 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 19e8a70ac22749b9bb6a392b989d1806; no voters:
I20260806 06:30:28.421850 19671 leader_election.cc:288] T 00000000000000000000000000000000 P 19e8a70ac22749b9bb6a392b989d1806 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 06:30:28.421896 19676 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 19e8a70ac22749b9bb6a392b989d1806 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 06:30:28.422062 19676 raft_consensus.cc:686] T 00000000000000000000000000000000 P 19e8a70ac22749b9bb6a392b989d1806 [term 1 LEADER]: Becoming Leader. State: Replica: 19e8a70ac22749b9bb6a392b989d1806, State: Running, Role: LEADER
I20260806 06:30:28.422117 19671 sys_catalog.cc:564] T 00000000000000000000000000000000 P 19e8a70ac22749b9bb6a392b989d1806 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 06:30:28.422186 19676 consensus_queue.cc:237] T 00000000000000000000000000000000 P 19e8a70ac22749b9bb6a392b989d1806 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19e8a70ac22749b9bb6a392b989d1806" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43899 } }
I20260806 06:30:28.422530 19676 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19e8a70ac22749b9bb6a392b989d1806 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 19e8a70ac22749b9bb6a392b989d1806. Latest consensus state: current_term: 1 leader_uuid: "19e8a70ac22749b9bb6a392b989d1806" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19e8a70ac22749b9bb6a392b989d1806" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43899 } } }
I20260806 06:30:28.422603 19676 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19e8a70ac22749b9bb6a392b989d1806 [sys.catalog]: This master's current role is: LEADER
I20260806 06:30:28.422576 19678 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19e8a70ac22749b9bb6a392b989d1806 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "19e8a70ac22749b9bb6a392b989d1806" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19e8a70ac22749b9bb6a392b989d1806" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43899 } } }
I20260806 06:30:28.422674 19678 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19e8a70ac22749b9bb6a392b989d1806 [sys.catalog]: This master's current role is: LEADER
I20260806 06:30:28.422808 19688 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 06:30:28.423548 19688 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 06:30:28.425189 19674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:28.415299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44767 (local address 127.0.0.1:43899)
0806 06:30:28.415455 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0806 06:30:28.415462 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 06:30:28.415487 (+ 25us) server_negotiation.cc:407] Connection header received
0806 06:30:28.415899 (+ 412us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:30:28.415906 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:30:28.415951 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:30:28.416069 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:30:28.416826 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:28.417729 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:28.418700 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:28.419105 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:28.419186 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:30:28.421685 (+ 2499us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:30:28.421689 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:30:28.421689 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 06:30:28.421701 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:30:28.423687 (+ 1986us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:30:28.424052 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:30:28.424055 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:30:28.424055 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:30:28.424169 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:30:28.424345 (+ 176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:30:28.424345 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:30:28.424346 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:30:28.424627 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:30:28.424760 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0806 06:30:28.424857 (+ 97us) server_negotiation.cc:299] Negotiation successful
0806 06:30:28.424985 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":67,"threads_started":1}
I20260806 06:30:28.425352 19688 catalog_manager.cc:1347] Generated new cluster ID: 999c4c17c57e4bcd89d603145aeca383
I20260806 06:30:28.425395 19688 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 06:30:28.426059 21554 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 06:30:28.432530 19688 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 06:30:28.433064 19688 catalog_manager.cc:1504] Loading token signing keys...
I20260806 06:30:28.435760 19688 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 19e8a70ac22749b9bb6a392b989d1806: Generated new TSK 0
I20260806 06:30:28.435918 19688 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 06:30:28.437073 19674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:28.427361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41057 (local address 127.0.0.1:43899)
0806 06:30:28.427394 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0806 06:30:28.427396 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:30:28.430472 (+ 3076us) server_negotiation.cc:407] Connection header received
0806 06:30:28.430526 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:30:28.430526 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:30:28.430535 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:30:28.430588 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:30:28.431237 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:28.431943 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:28.432951 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:28.433239 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:28.433302 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:30:28.434653 (+ 1351us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:30:28.434654 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:30:28.434655 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 06:30:28.434661 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:30:28.436161 (+ 1500us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:30:28.436508 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:30:28.436509 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:30:28.436509 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:30:28.436552 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:30:28.436700 (+ 148us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:30:28.436701 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:30:28.436701 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:30:28.436781 (+ 80us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:30:28.436843 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0806 06:30:28.436958 (+ 115us) server_negotiation.cc:299] Negotiation successful
0806 06:30:28.437019 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260806 06:30:28.438891 21554 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHhLhT/build/release/bin/kudu with pid 19576
2026-08-06T06:30:28Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1399 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785997581049705-21554-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 06:30:28 dist-test-slave-4tdj krb5kdc[19714](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 06:30:28 dist-test-slave-4tdj krb5kdc[19714](info): set up 2 sockets
Aug 06 06:30:28 dist-test-slave-4tdj krb5kdc[19714](info): commencing operation
krb5kdc: starting...
W20260806 06:30:29.565501 21554 mini_kdc.cc:121] Time spent starting KDC: real 1.099s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785997581049705-21554-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785997581049705-21554-0/krb5kdc/test-user.keytab.
Aug 06 06:30:29 dist-test-slave-4tdj krb5kdc[19714](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997829, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T06:30:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T06:30:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785997581049705-21554-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785997581049705-21554-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 06:30:29.683202 21554 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHhLhT/build/release/bin/kudu
/tmp/dist-test-taskFHhLhT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785997581049705-21554-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785997581049705-21554-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785997581049705-21554-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785997581049705-21554-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36285
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785997581049705-21554-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36285
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785997581049705-21554-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785997581049705-21554-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785997581049705-21554-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 06:30:29.764737 19729 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 06:30:29.764940 19729 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 06:30:29.764961 19729 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 06:30:29.766120 19729 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 06:30:29.766160 19729 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 06:30:29.766173 19729 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 06:30:29.766196 19729 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 06:30:29.766218 19729 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 06:30:29.767767 19729 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785997581049705-21554-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785997581049705-21554-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36285
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785997581049705-21554-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36285
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785997581049705-21554-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19729
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785997581049705-21554-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 05:11:02 UTC on ccycloud.quasar-hvzbbz.root.comops.site
build id 4569617
I20260806 06:30:29.768006 19729 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 06:30:29.768209 19729 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 06:30:29.772830 19740 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:30:29.772838 19738 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:30:29.772881 19742 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:30:29.773018 19729 server_base.cc:1034] running on GCE node
I20260806 06:30:29.773314 19729 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 06:30:29.773628 19729 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 06:30:29.774768 19729 hybrid_clock.cc:648] HybridClock initialized: now 1785997829774752 us; error 28 us; skew 500 ppm
Aug 06 06:30:29 dist-test-slave-4tdj krb5kdc[19714](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997829, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 06:30:29.777112 19729 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 06:30:29.777522 19729 webserver.cc:460] Webserver started at http://127.0.0.1:41531/ using document root <none> and password file <none>
I20260806 06:30:29.777733 19729 fs_manager.cc:359] Metadata directory not provided
I20260806 06:30:29.777781 19729 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 06:30:29.777907 19729 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 06:30:29.778978 19729 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785997581049705-21554-0/minicluster-data/master-0/data/instance:
uuid: "4e00262e121145efa026b613bb1c8bd9"
format_stamp: "Formatted at 2026-08-06 06:30:29 on dist-test-slave-4tdj"
server_key: "029551bfed0d18f39cd8c7bd7ea68f80"
server_key_iv: "b6052eb647681a2531da0437b6c8f4f6"
server_key_version: "encryptionkey@0"
I20260806 06:30:29.779315 19729 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785997581049705-21554-0/minicluster-data/master-0/wal/instance:
uuid: "4e00262e121145efa026b613bb1c8bd9"
format_stamp: "Formatted at 2026-08-06 06:30:29 on dist-test-slave-4tdj"
server_key: "029551bfed0d18f39cd8c7bd7ea68f80"
server_key_iv: "b6052eb647681a2531da0437b6c8f4f6"
server_key_version: "encryptionkey@0"
I20260806 06:30:29.780632 19729 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260806 06:30:29.781368 19755 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 06:30:29.781530 19729 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260806 06:30:29.781618 19729 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785997581049705-21554-0/minicluster-data/master-0/data,/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785997581049705-21554-0/minicluster-data/master-0/wal
uuid: "4e00262e121145efa026b613bb1c8bd9"
format_stamp: "Formatted at 2026-08-06 06:30:29 on dist-test-slave-4tdj"
server_key: "029551bfed0d18f39cd8c7bd7ea68f80"
server_key_iv: "b6052eb647681a2531da0437b6c8f4f6"
server_key_version: "encryptionkey@0"
I20260806 06:30:29.781684 19729 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785997581049705-21554-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785997581049705-21554-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785997581049705-21554-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 06:30:29.806651 19729 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 06:30:29.808914 19729 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 06:30:29.809078 19729 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 06:30:29.813104 19729 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36285
I20260806 06:30:29.813150 19827 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36285 every 8 connection(s)
I20260806 06:30:29.813494 19729 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785997581049705-21554-0/minicluster-data/master-0/data/info.pb
I20260806 06:30:29.814455 19828 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 06:30:29.816732 19828 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e00262e121145efa026b613bb1c8bd9: Bootstrap starting.
I20260806 06:30:29.817306 19828 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4e00262e121145efa026b613bb1c8bd9: Neither blocks nor log segments found. Creating new log.
I20260806 06:30:29.817557 19828 log.cc:824] T 00000000000000000000000000000000 P 4e00262e121145efa026b613bb1c8bd9: Log is configured to *not* fsync() on all Append() calls
I20260806 06:30:29.817873 21554 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHhLhT/build/release/bin/kudu as pid 19729
I20260806 06:30:29.817984 21554 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785997581049705-21554-0/minicluster-data/master-0/wal/instance
I20260806 06:30:29.818198 21554 external_mini_cluster.cc:1428] Setting key 28bf7b95c72732d9b6f2ed97548ca5aa
I20260806 06:30:29.818351 19828 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e00262e121145efa026b613bb1c8bd9: No bootstrap required, opened a new log
I20260806 06:30:29.820819 19828 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4e00262e121145efa026b613bb1c8bd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e00262e121145efa026b613bb1c8bd9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36285 } }
I20260806 06:30:29.820977 19828 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4e00262e121145efa026b613bb1c8bd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 06:30:29.821013 19828 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4e00262e121145efa026b613bb1c8bd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e00262e121145efa026b613bb1c8bd9, State: Initialized, Role: FOLLOWER
I20260806 06:30:29.821141 19828 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4e00262e121145efa026b613bb1c8bd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e00262e121145efa026b613bb1c8bd9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36285 } }
I20260806 06:30:29.821228 19828 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4e00262e121145efa026b613bb1c8bd9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 06:30:29.821266 19828 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4e00262e121145efa026b613bb1c8bd9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 06:30:29.821314 19828 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4e00262e121145efa026b613bb1c8bd9 [term 0 FOLLOWER]: Advancing to term 1
I20260806 06:30:29.821972 19828 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4e00262e121145efa026b613bb1c8bd9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e00262e121145efa026b613bb1c8bd9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36285 } }
I20260806 06:30:29.822062 19828 leader_election.cc:302] T 00000000000000000000000000000000 P 4e00262e121145efa026b613bb1c8bd9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4e00262e121145efa026b613bb1c8bd9; no voters:
I20260806 06:30:29.822227 19828 leader_election.cc:288] T 00000000000000000000000000000000 P 4e00262e121145efa026b613bb1c8bd9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 06:30:29.822259 19835 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4e00262e121145efa026b613bb1c8bd9 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 06:30:29.822422 19835 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4e00262e121145efa026b613bb1c8bd9 [term 1 LEADER]: Becoming Leader. State: Replica: 4e00262e121145efa026b613bb1c8bd9, State: Running, Role: LEADER
I20260806 06:30:29.822520 19828 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4e00262e121145efa026b613bb1c8bd9 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 06:30:29.822598 19835 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4e00262e121145efa026b613bb1c8bd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e00262e121145efa026b613bb1c8bd9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36285 } }
I20260806 06:30:29.822976 19835 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e00262e121145efa026b613bb1c8bd9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4e00262e121145efa026b613bb1c8bd9. Latest consensus state: current_term: 1 leader_uuid: "4e00262e121145efa026b613bb1c8bd9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e00262e121145efa026b613bb1c8bd9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36285 } } }
I20260806 06:30:29.823104 19835 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e00262e121145efa026b613bb1c8bd9 [sys.catalog]: This master's current role is: LEADER
I20260806 06:30:29.823028 19836 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e00262e121145efa026b613bb1c8bd9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4e00262e121145efa026b613bb1c8bd9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e00262e121145efa026b613bb1c8bd9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36285 } } }
I20260806 06:30:29.823307 19836 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e00262e121145efa026b613bb1c8bd9 [sys.catalog]: This master's current role is: LEADER
I20260806 06:30:29.823408 19847 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 06:30:29.823951 19847 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 06:30:29.823998 19832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:29.818779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55917 (local address 127.0.0.1:36285)
0806 06:30:29.818952 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0806 06:30:29.818960 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 06:30:29.818987 (+ 27us) server_negotiation.cc:407] Connection header received
0806 06:30:29.819607 (+ 620us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:30:29.819614 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:30:29.819663 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:30:29.819775 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:30:29.819997 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:29.820852 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:29.821894 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:29.822437 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:29.822585 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:30:29.823247 (+ 662us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:30:29.823252 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:30:29.823252 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 06:30:29.823453 (+ 201us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:30:29.823518 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 06:30:29.823535 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:30:29.823595 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0806 06:30:29.823694 (+ 99us) server_negotiation.cc:299] Negotiation successful
0806 06:30:29.823761 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":70,"threads_started":1}
I20260806 06:30:29.824959 21554 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 06:30:29.826452 19847 catalog_manager.cc:1347] Generated new cluster ID: a9ad96186a054d60879dfd23a6669833
I20260806 06:30:29.826505 19847 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 06:30:29.832818 19832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:29.826238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56635 (local address 127.0.0.1:36285)
0806 06:30:29.826270 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0806 06:30:29.826272 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:30:29.830367 (+ 4095us) server_negotiation.cc:407] Connection header received
0806 06:30:29.830391 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:30:29.830391 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:30:29.830402 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:30:29.830457 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:30:29.830598 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:29.831074 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:29.831749 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:29.831935 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:29.831987 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:30:29.832613 (+ 626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:30:29.832614 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:30:29.832615 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 06:30:29.832618 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:30:29.832631 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 06:30:29.832634 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:30:29.832675 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0806 06:30:29.832727 (+ 52us) server_negotiation.cc:299] Negotiation successful
0806 06:30:29.832748 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260806 06:30:29.836939 19847 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 06:30:29.837437 19847 catalog_manager.cc:1504] Loading token signing keys...
I20260806 06:30:29.842154 19847 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4e00262e121145efa026b613bb1c8bd9: Generated new TSK 0
I20260806 06:30:29.842353 19847 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 06:30:29.844915 21554 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHhLhT/build/release/bin/kudu with pid 19729
2026-08-06T06:30:29Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1406 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785997581049705-21554-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 06:30:29 dist-test-slave-4tdj krb5kdc[19872](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 06:30:29 dist-test-slave-4tdj krb5kdc[19872](info): set up 2 sockets
Aug 06 06:30:29 dist-test-slave-4tdj krb5kdc[19872](info): commencing operation
krb5kdc: starting...
W20260806 06:30:30.975423 21554 mini_kdc.cc:121] Time spent starting KDC: real 1.102s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785997581049705-21554-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785997581049705-21554-0/krb5kdc/test-user.keytab.
Aug 06 06:30:31 dist-test-slave-4tdj krb5kdc[19872](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997831, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T06:30:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T06:30:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785997581049705-21554-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785997581049705-21554-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 06:30:31.097376 21554 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHhLhT/build/release/bin/kudu
/tmp/dist-test-taskFHhLhT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785997581049705-21554-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785997581049705-21554-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785997581049705-21554-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785997581049705-21554-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43071
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785997581049705-21554-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43071
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785997581049705-21554-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785997581049705-21554-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785997581049705-21554-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 06:30:31.178714 19888 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 06:30:31.178926 19888 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 06:30:31.178957 19888 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 06:30:31.180096 19888 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 06:30:31.180140 19888 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 06:30:31.180158 19888 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 06:30:31.180173 19888 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 06:30:31.180188 19888 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 06:30:31.181679 19888 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785997581049705-21554-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785997581049705-21554-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43071
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785997581049705-21554-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43071
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785997581049705-21554-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19888
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785997581049705-21554-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 05:11:02 UTC on ccycloud.quasar-hvzbbz.root.comops.site
build id 4569617
I20260806 06:30:31.181921 19888 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 06:30:31.182126 19888 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 06:30:31.187309 19897 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:30:31.187479 19894 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:30:31.187309 19893 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:30:31.187732 19888 server_base.cc:1034] running on GCE node
I20260806 06:30:31.187974 19888 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 06:30:31.188295 19888 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 06:30:31.189451 19888 hybrid_clock.cc:648] HybridClock initialized: now 1785997831189431 us; error 35 us; skew 500 ppm
Aug 06 06:30:31 dist-test-slave-4tdj krb5kdc[19872](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997831, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 06:30:31.191936 19888 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 06:30:31.192400 19888 webserver.cc:460] Webserver started at http://127.0.0.1:33279/ using document root <none> and password file <none>
I20260806 06:30:31.192615 19888 fs_manager.cc:359] Metadata directory not provided
I20260806 06:30:31.192662 19888 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 06:30:31.192792 19888 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 06:30:31.193887 19888 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785997581049705-21554-0/minicluster-data/master-0/data/instance:
uuid: "7610ab7c77e64e798881841a619f41e1"
format_stamp: "Formatted at 2026-08-06 06:30:31 on dist-test-slave-4tdj"
server_key: "11f070bbb7eeb96c022e7a389ff9ccb6"
server_key_iv: "aa3055df082437986caf3db180c70a64"
server_key_version: "encryptionkey@0"
I20260806 06:30:31.194231 19888 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785997581049705-21554-0/minicluster-data/master-0/wal/instance:
uuid: "7610ab7c77e64e798881841a619f41e1"
format_stamp: "Formatted at 2026-08-06 06:30:31 on dist-test-slave-4tdj"
server_key: "11f070bbb7eeb96c022e7a389ff9ccb6"
server_key_iv: "aa3055df082437986caf3db180c70a64"
server_key_version: "encryptionkey@0"
I20260806 06:30:31.195525 19888 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260806 06:30:31.196242 19907 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 06:30:31.196396 19888 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260806 06:30:31.196486 19888 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785997581049705-21554-0/minicluster-data/master-0/data,/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785997581049705-21554-0/minicluster-data/master-0/wal
uuid: "7610ab7c77e64e798881841a619f41e1"
format_stamp: "Formatted at 2026-08-06 06:30:31 on dist-test-slave-4tdj"
server_key: "11f070bbb7eeb96c022e7a389ff9ccb6"
server_key_iv: "aa3055df082437986caf3db180c70a64"
server_key_version: "encryptionkey@0"
I20260806 06:30:31.196557 19888 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785997581049705-21554-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785997581049705-21554-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785997581049705-21554-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 06:30:31.199892 19888 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 06:30:31.201745 19888 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 06:30:31.201880 19888 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 06:30:31.205749 19888 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43071
I20260806 06:30:31.205791 19979 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43071 every 8 connection(s)
I20260806 06:30:31.206118 19888 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785997581049705-21554-0/minicluster-data/master-0/data/info.pb
I20260806 06:30:31.207158 19980 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 06:30:31.209385 19980 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7610ab7c77e64e798881841a619f41e1: Bootstrap starting.
I20260806 06:30:31.209929 19980 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7610ab7c77e64e798881841a619f41e1: Neither blocks nor log segments found. Creating new log.
I20260806 06:30:31.210162 19980 log.cc:824] T 00000000000000000000000000000000 P 7610ab7c77e64e798881841a619f41e1: Log is configured to *not* fsync() on all Append() calls
I20260806 06:30:31.210839 19980 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7610ab7c77e64e798881841a619f41e1: No bootstrap required, opened a new log
I20260806 06:30:31.212380 19980 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7610ab7c77e64e798881841a619f41e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7610ab7c77e64e798881841a619f41e1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43071 } }
I20260806 06:30:31.212512 21554 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHhLhT/build/release/bin/kudu as pid 19888
I20260806 06:30:31.212569 19980 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7610ab7c77e64e798881841a619f41e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 06:30:31.212621 21554 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785997581049705-21554-0/minicluster-data/master-0/wal/instance
I20260806 06:30:31.212615 19980 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7610ab7c77e64e798881841a619f41e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7610ab7c77e64e798881841a619f41e1, State: Initialized, Role: FOLLOWER
I20260806 06:30:31.212766 19980 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7610ab7c77e64e798881841a619f41e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7610ab7c77e64e798881841a619f41e1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43071 } }
I20260806 06:30:31.212823 21554 external_mini_cluster.cc:1428] Setting key 3bda5a919dc4934628045012b5d3e69c
I20260806 06:30:31.212846 19980 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7610ab7c77e64e798881841a619f41e1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 06:30:31.212885 19980 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7610ab7c77e64e798881841a619f41e1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 06:30:31.212934 19980 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7610ab7c77e64e798881841a619f41e1 [term 0 FOLLOWER]: Advancing to term 1
I20260806 06:30:31.213872 19980 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7610ab7c77e64e798881841a619f41e1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7610ab7c77e64e798881841a619f41e1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43071 } }
I20260806 06:30:31.213986 19980 leader_election.cc:302] T 00000000000000000000000000000000 P 7610ab7c77e64e798881841a619f41e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7610ab7c77e64e798881841a619f41e1; no voters:
I20260806 06:30:31.214147 19980 leader_election.cc:288] T 00000000000000000000000000000000 P 7610ab7c77e64e798881841a619f41e1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 06:30:31.214188 19985 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7610ab7c77e64e798881841a619f41e1 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 06:30:31.214350 19985 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7610ab7c77e64e798881841a619f41e1 [term 1 LEADER]: Becoming Leader. State: Replica: 7610ab7c77e64e798881841a619f41e1, State: Running, Role: LEADER
I20260806 06:30:31.214411 19980 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7610ab7c77e64e798881841a619f41e1 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 06:30:31.214445 19985 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7610ab7c77e64e798881841a619f41e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7610ab7c77e64e798881841a619f41e1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43071 } }
I20260806 06:30:31.214951 19986 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7610ab7c77e64e798881841a619f41e1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7610ab7c77e64e798881841a619f41e1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7610ab7c77e64e798881841a619f41e1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43071 } } }
I20260806 06:30:31.215039 19986 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7610ab7c77e64e798881841a619f41e1 [sys.catalog]: This master's current role is: LEADER
I20260806 06:30:31.215260 19997 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 06:30:31.215252 19988 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7610ab7c77e64e798881841a619f41e1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7610ab7c77e64e798881841a619f41e1. Latest consensus state: current_term: 1 leader_uuid: "7610ab7c77e64e798881841a619f41e1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7610ab7c77e64e798881841a619f41e1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43071 } } }
I20260806 06:30:31.215550 19988 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7610ab7c77e64e798881841a619f41e1 [sys.catalog]: This master's current role is: LEADER
I20260806 06:30:31.215834 19997 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 06:30:31.217281 19997 catalog_manager.cc:1347] Generated new cluster ID: 1a44c28e14ca45b297a839b1eba6d87d
I20260806 06:30:31.217324 19997 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 06:30:31.222049 19997 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 06:30:31.222218 19997 catalog_manager.cc:1504] Loading token signing keys...
I20260806 06:30:31.222718 19984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:31.213319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56437 (local address 127.0.0.1:43071)
0806 06:30:31.213506 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0806 06:30:31.213515 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 06:30:31.213550 (+ 35us) server_negotiation.cc:407] Connection header received
0806 06:30:31.213644 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:30:31.213649 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:30:31.213672 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:30:31.213778 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:30:31.222069 (+ 8291us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:30:31.222073 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:30:31.222073 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 06:30:31.222080 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:30:31.222132 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 06:30:31.222148 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:30:31.222227 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0806 06:30:31.222493 (+ 266us) server_negotiation.cc:299] Negotiation successful
0806 06:30:31.222536 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":92,"threads_started":1}
I20260806 06:30:31.223734 21554 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 06:30:31.226440 19984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:31.225276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46325 (local address 127.0.0.1:43071)
0806 06:30:31.225320 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0806 06:30:31.225322 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:30:31.225482 (+ 160us) server_negotiation.cc:407] Connection header received
0806 06:30:31.225564 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:30:31.225565 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:30:31.225570 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:30:31.225595 (+ 25us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:30:31.226202 (+ 607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:30:31.226203 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:30:31.226203 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 06:30:31.226207 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:30:31.226220 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 06:30:31.226224 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:30:31.226248 (+ 24us) server_negotiation.cc:1035] Waiting for connection context
0806 06:30:31.226357 (+ 109us) server_negotiation.cc:299] Negotiation successful
0806 06:30:31.226369 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260806 06:30:31.232208 19997 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7610ab7c77e64e798881841a619f41e1: Generated new TSK 0
I20260806 06:30:31.232389 19997 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 06:30:31.237012 21554 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHhLhT/build/release/bin/kudu with pid 19888
2026-08-06T06:30:31Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1389 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785997581049705-21554-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 06:30:31 dist-test-slave-4tdj krb5kdc[20017](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 06:30:31 dist-test-slave-4tdj krb5kdc[20017](info): set up 2 sockets
Aug 06 06:30:31 dist-test-slave-4tdj krb5kdc[20017](info): commencing operation
krb5kdc: starting...
W20260806 06:30:32.365219 21554 mini_kdc.cc:121] Time spent starting KDC: real 1.103s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785997581049705-21554-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785997581049705-21554-0/krb5kdc/test-user.keytab.
Aug 06 06:30:32 dist-test-slave-4tdj krb5kdc[20017](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997832, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T06:30:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T06:30:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785997581049705-21554-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785997581049705-21554-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 06:30:32.480252 21554 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHhLhT/build/release/bin/kudu
/tmp/dist-test-taskFHhLhT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785997581049705-21554-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785997581049705-21554-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785997581049705-21554-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785997581049705-21554-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34243
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785997581049705-21554-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34243
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785997581049705-21554-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785997581049705-21554-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785997581049705-21554-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 06:30:32.559906 20036 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 06:30:32.560101 20036 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 06:30:32.560122 20036 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 06:30:32.561242 20036 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 06:30:32.561285 20036 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 06:30:32.561297 20036 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 06:30:32.561307 20036 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 06:30:32.561321 20036 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 06:30:32.562896 20036 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785997581049705-21554-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785997581049705-21554-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34243
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785997581049705-21554-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34243
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785997581049705-21554-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20036
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785997581049705-21554-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 05:11:02 UTC on ccycloud.quasar-hvzbbz.root.comops.site
build id 4569617
I20260806 06:30:32.563118 20036 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 06:30:32.563331 20036 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 06:30:32.568003 20045 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:30:32.568050 20047 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:30:32.568009 20044 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:30:32.568213 20036 server_base.cc:1034] running on GCE node
I20260806 06:30:32.568445 20036 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 06:30:32.568732 20036 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 06:30:32.569877 20036 hybrid_clock.cc:648] HybridClock initialized: now 1785997832569864 us; error 29 us; skew 500 ppm
Aug 06 06:30:32 dist-test-slave-4tdj krb5kdc[20017](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997832, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 06:30:32.572225 20036 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 06:30:32.572662 20036 webserver.cc:460] Webserver started at http://127.0.0.1:39919/ using document root <none> and password file <none>
I20260806 06:30:32.572878 20036 fs_manager.cc:359] Metadata directory not provided
I20260806 06:30:32.572925 20036 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 06:30:32.573048 20036 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 06:30:32.574054 20036 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785997581049705-21554-0/minicluster-data/master-0/data/instance:
uuid: "06663b2384f44caab4c3b15fba4f8643"
format_stamp: "Formatted at 2026-08-06 06:30:32 on dist-test-slave-4tdj"
server_key: "a7c3d275065f63ce074c2c928afa87f5"
server_key_iv: "894e440259e969459320a7e27dcb1dcb"
server_key_version: "encryptionkey@0"
I20260806 06:30:32.574412 20036 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785997581049705-21554-0/minicluster-data/master-0/wal/instance:
uuid: "06663b2384f44caab4c3b15fba4f8643"
format_stamp: "Formatted at 2026-08-06 06:30:32 on dist-test-slave-4tdj"
server_key: "a7c3d275065f63ce074c2c928afa87f5"
server_key_iv: "894e440259e969459320a7e27dcb1dcb"
server_key_version: "encryptionkey@0"
I20260806 06:30:32.575625 20036 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260806 06:30:32.576301 20056 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 06:30:32.576465 20036 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260806 06:30:32.576550 20036 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785997581049705-21554-0/minicluster-data/master-0/data,/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785997581049705-21554-0/minicluster-data/master-0/wal
uuid: "06663b2384f44caab4c3b15fba4f8643"
format_stamp: "Formatted at 2026-08-06 06:30:32 on dist-test-slave-4tdj"
server_key: "a7c3d275065f63ce074c2c928afa87f5"
server_key_iv: "894e440259e969459320a7e27dcb1dcb"
server_key_version: "encryptionkey@0"
I20260806 06:30:32.576614 20036 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785997581049705-21554-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785997581049705-21554-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785997581049705-21554-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 06:30:32.589869 20036 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 06:30:32.591547 20036 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 06:30:32.591678 20036 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 06:30:32.595551 20036 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34243
I20260806 06:30:32.595587 20130 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34243 every 8 connection(s)
I20260806 06:30:32.595901 20036 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785997581049705-21554-0/minicluster-data/master-0/data/info.pb
I20260806 06:30:32.596722 20131 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 06:30:32.598894 20131 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 06663b2384f44caab4c3b15fba4f8643: Bootstrap starting.
I20260806 06:30:32.599416 20131 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 06663b2384f44caab4c3b15fba4f8643: Neither blocks nor log segments found. Creating new log.
I20260806 06:30:32.599644 20131 log.cc:824] T 00000000000000000000000000000000 P 06663b2384f44caab4c3b15fba4f8643: Log is configured to *not* fsync() on all Append() calls
I20260806 06:30:32.600265 20131 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 06663b2384f44caab4c3b15fba4f8643: No bootstrap required, opened a new log
I20260806 06:30:32.601770 20131 raft_consensus.cc:348] T 00000000000000000000000000000000 P 06663b2384f44caab4c3b15fba4f8643 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06663b2384f44caab4c3b15fba4f8643" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34243 } }
I20260806 06:30:32.601903 20131 raft_consensus.cc:374] T 00000000000000000000000000000000 P 06663b2384f44caab4c3b15fba4f8643 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 06:30:32.601924 20131 raft_consensus.cc:729] T 00000000000000000000000000000000 P 06663b2384f44caab4c3b15fba4f8643 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06663b2384f44caab4c3b15fba4f8643, State: Initialized, Role: FOLLOWER
I20260806 06:30:32.602036 20131 consensus_queue.cc:260] T 00000000000000000000000000000000 P 06663b2384f44caab4c3b15fba4f8643 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06663b2384f44caab4c3b15fba4f8643" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34243 } }
I20260806 06:30:32.602087 20131 raft_consensus.cc:388] T 00000000000000000000000000000000 P 06663b2384f44caab4c3b15fba4f8643 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 06:30:32.602106 20131 raft_consensus.cc:482] T 00000000000000000000000000000000 P 06663b2384f44caab4c3b15fba4f8643 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 06:30:32.602136 20131 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 06663b2384f44caab4c3b15fba4f8643 [term 0 FOLLOWER]: Advancing to term 1
I20260806 06:30:32.602656 20131 raft_consensus.cc:504] T 00000000000000000000000000000000 P 06663b2384f44caab4c3b15fba4f8643 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06663b2384f44caab4c3b15fba4f8643" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34243 } }
I20260806 06:30:32.602742 20131 leader_election.cc:302] T 00000000000000000000000000000000 P 06663b2384f44caab4c3b15fba4f8643 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 06663b2384f44caab4c3b15fba4f8643; no voters:
I20260806 06:30:32.602859 20131 leader_election.cc:288] T 00000000000000000000000000000000 P 06663b2384f44caab4c3b15fba4f8643 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 06:30:32.602910 20135 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 06663b2384f44caab4c3b15fba4f8643 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 06:30:32.603076 20131 sys_catalog.cc:564] T 00000000000000000000000000000000 P 06663b2384f44caab4c3b15fba4f8643 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 06:30:32.603084 20135 raft_consensus.cc:686] T 00000000000000000000000000000000 P 06663b2384f44caab4c3b15fba4f8643 [term 1 LEADER]: Becoming Leader. State: Replica: 06663b2384f44caab4c3b15fba4f8643, State: Running, Role: LEADER
I20260806 06:30:32.603273 20135 consensus_queue.cc:237] T 00000000000000000000000000000000 P 06663b2384f44caab4c3b15fba4f8643 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06663b2384f44caab4c3b15fba4f8643" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34243 } }
I20260806 06:30:32.603852 20137 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06663b2384f44caab4c3b15fba4f8643 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 06663b2384f44caab4c3b15fba4f8643. Latest consensus state: current_term: 1 leader_uuid: "06663b2384f44caab4c3b15fba4f8643" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06663b2384f44caab4c3b15fba4f8643" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34243 } } }
I20260806 06:30:32.603843 20136 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06663b2384f44caab4c3b15fba4f8643 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "06663b2384f44caab4c3b15fba4f8643" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06663b2384f44caab4c3b15fba4f8643" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34243 } } }
I20260806 06:30:32.603940 20137 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06663b2384f44caab4c3b15fba4f8643 [sys.catalog]: This master's current role is: LEADER
I20260806 06:30:32.603945 20136 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06663b2384f44caab4c3b15fba4f8643 [sys.catalog]: This master's current role is: LEADER
I20260806 06:30:32.604192 20146 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 06:30:32.604794 21554 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHhLhT/build/release/bin/kudu as pid 20036
I20260806 06:30:32.604907 21554 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785997581049705-21554-0/minicluster-data/master-0/wal/instance
I20260806 06:30:32.605049 21554 external_mini_cluster.cc:1428] Setting key 8de9f85f2c7549e42d6606b8a0d0addf
I20260806 06:30:32.605368 20146 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 06:30:32.607934 20146 catalog_manager.cc:1347] Generated new cluster ID: 2d8505ce9902446b8ca5739895669f11
I20260806 06:30:32.607990 20146 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 06 06:30:32 dist-test-slave-4tdj krb5kdc[20017](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997832, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260806 06:30:32.615808 20156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:32.605685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.56:59921 (local address 127.0.0.1:34243)
0806 06:30:32.605858 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0806 06:30:32.605867 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 06:30:32.605899 (+ 32us) server_negotiation.cc:407] Connection header received
0806 06:30:32.606025 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:30:32.606032 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:30:32.606078 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:30:32.606192 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:30:32.606899 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:32.607809 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:32.608826 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:32.609460 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:32.609585 (+ 125us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:30:32.612226 (+ 2641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:30:32.612231 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:30:32.612231 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 06:30:32.612242 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:30:32.614045 (+ 1803us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:30:32.614607 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:30:32.614609 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:30:32.614610 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:30:32.614749 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:30:32.614965 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:30:32.614965 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:30:32.614966 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:30:32.615243 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:30:32.615381 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0806 06:30:32.615496 (+ 115us) server_negotiation.cc:299] Negotiation successful
0806 06:30:32.615591 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":82,"threads_started":1}
I20260806 06:30:32.616797 21554 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 06:30:32.628307 20146 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 06:30:32.629000 20146 catalog_manager.cc:1504] Loading token signing keys...
I20260806 06:30:32.632285 20146 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 06663b2384f44caab4c3b15fba4f8643: Generated new TSK 0
I20260806 06:30:32.632521 20146 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 06:30:32.633285 20156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:32.618051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.56:45881 (local address 127.0.0.1:34243)
0806 06:30:32.618086 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0806 06:30:32.618088 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:30:32.626460 (+ 8372us) server_negotiation.cc:407] Connection header received
0806 06:30:32.626513 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:30:32.626514 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:30:32.626526 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:30:32.626591 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:30:32.627210 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:32.627998 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:32.628947 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:32.629283 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:32.629347 (+ 64us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:30:32.630741 (+ 1394us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:30:32.630743 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:30:32.630743 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 06:30:32.630750 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:30:32.632134 (+ 1384us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:30:32.632535 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:30:32.632537 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:30:32.632537 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:30:32.632598 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:30:32.632808 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:30:32.632809 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:30:32.632809 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:30:32.632930 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:30:32.633023 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0806 06:30:32.633126 (+ 103us) server_negotiation.cc:299] Negotiation successful
0806 06:30:32.633196 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260806 06:30:32.634969 21554 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHhLhT/build/release/bin/kudu with pid 20036
W20260806 06:30:32.646111 20015 connection.cc:535] client connection to 127.0.0.1:34243 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34243): Connection reset by peer (error 104)
2026-08-06T06:30:32Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1397 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785997581049705-21554-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 06:30:32 dist-test-slave-4tdj krb5kdc[20168](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 06:30:32 dist-test-slave-4tdj krb5kdc[20168](info): set up 2 sockets
Aug 06 06:30:32 dist-test-slave-4tdj krb5kdc[20168](info): commencing operation
krb5kdc: starting...
W20260806 06:30:33.762701 21554 mini_kdc.cc:121] Time spent starting KDC: real 1.103s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785997581049705-21554-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785997581049705-21554-0/krb5kdc/test-user.keytab.
Aug 06 06:30:33 dist-test-slave-4tdj krb5kdc[20168](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997833, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T06:30:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T06:30:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785997581049705-21554-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785997581049705-21554-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 06:30:33.876849 21554 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHhLhT/build/release/bin/kudu
/tmp/dist-test-taskFHhLhT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785997581049705-21554-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785997581049705-21554-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785997581049705-21554-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785997581049705-21554-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46391
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785997581049705-21554-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46391
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785997581049705-21554-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785997581049705-21554-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785997581049705-21554-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 06:30:33.956912 20188 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 06:30:33.957106 20188 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 06:30:33.957127 20188 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 06:30:33.958199 20188 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 06:30:33.958230 20188 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 06:30:33.958242 20188 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 06:30:33.958253 20188 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 06:30:33.958267 20188 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 06:30:33.959708 20188 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785997581049705-21554-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785997581049705-21554-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46391
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785997581049705-21554-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46391
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785997581049705-21554-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20188
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785997581049705-21554-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 05:11:02 UTC on ccycloud.quasar-hvzbbz.root.comops.site
build id 4569617
I20260806 06:30:33.959914 20188 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 06:30:33.960122 20188 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 06:30:33.964650 20202 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:30:33.964650 20195 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:30:33.964646 20197 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:30:33.964679 20188 server_base.cc:1034] running on GCE node
I20260806 06:30:33.965124 20188 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 06:30:33.965384 20188 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 06:30:33.966509 20188 hybrid_clock.cc:648] HybridClock initialized: now 1785997833966493 us; error 31 us; skew 500 ppm
Aug 06 06:30:33 dist-test-slave-4tdj krb5kdc[20168](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997833, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 06:30:33.968717 20188 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 06:30:33.969127 20188 webserver.cc:460] Webserver started at http://127.0.0.1:45415/ using document root <none> and password file <none>
I20260806 06:30:33.969333 20188 fs_manager.cc:359] Metadata directory not provided
I20260806 06:30:33.969376 20188 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 06:30:33.969501 20188 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 06:30:33.970544 20188 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785997581049705-21554-0/minicluster-data/master-0/data/instance:
uuid: "c69c0d74ed1144c199deb95be6c914b5"
format_stamp: "Formatted at 2026-08-06 06:30:33 on dist-test-slave-4tdj"
server_key: "2131093981e5dc59c2b6064df3e29a9f"
server_key_iv: "bb8a76fd7fe3573783849da9cea9068e"
server_key_version: "encryptionkey@0"
I20260806 06:30:33.970851 20188 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785997581049705-21554-0/minicluster-data/master-0/wal/instance:
uuid: "c69c0d74ed1144c199deb95be6c914b5"
format_stamp: "Formatted at 2026-08-06 06:30:33 on dist-test-slave-4tdj"
server_key: "2131093981e5dc59c2b6064df3e29a9f"
server_key_iv: "bb8a76fd7fe3573783849da9cea9068e"
server_key_version: "encryptionkey@0"
I20260806 06:30:33.972023 20188 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260806 06:30:33.972613 20212 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 06:30:33.972741 20188 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260806 06:30:33.972817 20188 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785997581049705-21554-0/minicluster-data/master-0/data,/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785997581049705-21554-0/minicluster-data/master-0/wal
uuid: "c69c0d74ed1144c199deb95be6c914b5"
format_stamp: "Formatted at 2026-08-06 06:30:33 on dist-test-slave-4tdj"
server_key: "2131093981e5dc59c2b6064df3e29a9f"
server_key_iv: "bb8a76fd7fe3573783849da9cea9068e"
server_key_version: "encryptionkey@0"
I20260806 06:30:33.972877 20188 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785997581049705-21554-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785997581049705-21554-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785997581049705-21554-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 06:30:33.984784 20188 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 06:30:33.986338 20188 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 06:30:33.986454 20188 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 06:30:33.990067 20188 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46391
I20260806 06:30:33.990105 20278 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46391 every 8 connection(s)
I20260806 06:30:33.990420 20188 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785997581049705-21554-0/minicluster-data/master-0/data/info.pb
I20260806 06:30:33.991286 20280 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 06:30:33.991532 21554 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHhLhT/build/release/bin/kudu as pid 20188
I20260806 06:30:33.991660 21554 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785997581049705-21554-0/minicluster-data/master-0/wal/instance
I20260806 06:30:33.991853 21554 external_mini_cluster.cc:1428] Setting key 0b1b2313abcff673e89c2c67d9c8b0b5
I20260806 06:30:33.994415 20280 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c69c0d74ed1144c199deb95be6c914b5: Bootstrap starting.
I20260806 06:30:33.995110 20280 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c69c0d74ed1144c199deb95be6c914b5: Neither blocks nor log segments found. Creating new log.
I20260806 06:30:33.995385 20280 log.cc:824] T 00000000000000000000000000000000 P c69c0d74ed1144c199deb95be6c914b5: Log is configured to *not* fsync() on all Append() calls
I20260806 06:30:33.996098 20280 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c69c0d74ed1144c199deb95be6c914b5: No bootstrap required, opened a new log
I20260806 06:30:33.997581 20283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:33.992331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.56:51351 (local address 127.0.0.1:46391)
0806 06:30:33.992534 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0806 06:30:33.992544 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 06:30:33.992580 (+ 36us) server_negotiation.cc:407] Connection header received
0806 06:30:33.993083 (+ 503us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:30:33.993093 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:30:33.993157 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:30:33.993318 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:30:33.993492 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:33.994844 (+ 1352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:33.995685 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:33.996169 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:33.996276 (+ 107us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:30:33.996984 (+ 708us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:30:33.996990 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:30:33.996990 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 06:30:33.997154 (+ 164us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:30:33.997185 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 06:30:33.997195 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:30:33.997275 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0806 06:30:33.997382 (+ 107us) server_negotiation.cc:299] Negotiation successful
0806 06:30:33.997411 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":73,"threads_started":1}
I20260806 06:30:33.997951 20280 raft_consensus.cc:348] T 00000000000000000000000000000000 P c69c0d74ed1144c199deb95be6c914b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c69c0d74ed1144c199deb95be6c914b5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46391 } }
I20260806 06:30:33.998100 20280 raft_consensus.cc:374] T 00000000000000000000000000000000 P c69c0d74ed1144c199deb95be6c914b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 06:30:33.998183 20280 raft_consensus.cc:729] T 00000000000000000000000000000000 P c69c0d74ed1144c199deb95be6c914b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c69c0d74ed1144c199deb95be6c914b5, State: Initialized, Role: FOLLOWER
I20260806 06:30:33.998329 20280 consensus_queue.cc:260] T 00000000000000000000000000000000 P c69c0d74ed1144c199deb95be6c914b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c69c0d74ed1144c199deb95be6c914b5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46391 } }
I20260806 06:30:33.998418 20280 raft_consensus.cc:388] T 00000000000000000000000000000000 P c69c0d74ed1144c199deb95be6c914b5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 06:30:33.998456 20280 raft_consensus.cc:482] T 00000000000000000000000000000000 P c69c0d74ed1144c199deb95be6c914b5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 06:30:33.998513 20280 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c69c0d74ed1144c199deb95be6c914b5 [term 0 FOLLOWER]: Advancing to term 1
I20260806 06:30:33.998603 21554 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 06:30:33.999069 20280 raft_consensus.cc:504] T 00000000000000000000000000000000 P c69c0d74ed1144c199deb95be6c914b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c69c0d74ed1144c199deb95be6c914b5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46391 } }
I20260806 06:30:33.999166 20280 leader_election.cc:302] T 00000000000000000000000000000000 P c69c0d74ed1144c199deb95be6c914b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c69c0d74ed1144c199deb95be6c914b5; no voters:
I20260806 06:30:33.999313 20280 leader_election.cc:288] T 00000000000000000000000000000000 P c69c0d74ed1144c199deb95be6c914b5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 06:30:33.999368 20287 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c69c0d74ed1144c199deb95be6c914b5 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 06:30:33.999564 20280 sys_catalog.cc:564] T 00000000000000000000000000000000 P c69c0d74ed1144c199deb95be6c914b5 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 06:30:33.999563 20287 raft_consensus.cc:686] T 00000000000000000000000000000000 P c69c0d74ed1144c199deb95be6c914b5 [term 1 LEADER]: Becoming Leader. State: Replica: c69c0d74ed1144c199deb95be6c914b5, State: Running, Role: LEADER
I20260806 06:30:33.999792 20287 consensus_queue.cc:237] T 00000000000000000000000000000000 P c69c0d74ed1144c199deb95be6c914b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c69c0d74ed1144c199deb95be6c914b5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46391 } }
I20260806 06:30:34.000168 20289 sys_catalog.cc:455] T 00000000000000000000000000000000 P c69c0d74ed1144c199deb95be6c914b5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c69c0d74ed1144c199deb95be6c914b5. Latest consensus state: current_term: 1 leader_uuid: "c69c0d74ed1144c199deb95be6c914b5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c69c0d74ed1144c199deb95be6c914b5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46391 } } }
I20260806 06:30:34.000175 20288 sys_catalog.cc:455] T 00000000000000000000000000000000 P c69c0d74ed1144c199deb95be6c914b5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c69c0d74ed1144c199deb95be6c914b5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c69c0d74ed1144c199deb95be6c914b5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46391 } } }
I20260806 06:30:34.000250 20289 sys_catalog.cc:458] T 00000000000000000000000000000000 P c69c0d74ed1144c199deb95be6c914b5 [sys.catalog]: This master's current role is: LEADER
I20260806 06:30:34.000294 20288 sys_catalog.cc:458] T 00000000000000000000000000000000 P c69c0d74ed1144c199deb95be6c914b5 [sys.catalog]: This master's current role is: LEADER
I20260806 06:30:34.000465 20307 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 06:30:34.000941 20307 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 06:30:34.002599 20307 catalog_manager.cc:1347] Generated new cluster ID: 1c72a4186bb442d99a9c1a26684ffa66
I20260806 06:30:34.002645 20307 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 06:30:34.004940 20283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:34.001137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.56:54757 (local address 127.0.0.1:46391)
0806 06:30:34.001168 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0806 06:30:34.001170 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:30:34.001453 (+ 283us) server_negotiation.cc:407] Connection header received
0806 06:30:34.001515 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:30:34.001516 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:30:34.001527 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:30:34.001574 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:30:34.001766 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:34.002469 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:34.003480 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:34.003798 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:34.003859 (+ 61us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:30:34.004582 (+ 723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:30:34.004584 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:30:34.004584 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 06:30:34.004589 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:30:34.004602 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 06:30:34.004606 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:30:34.004652 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0806 06:30:34.004855 (+ 203us) server_negotiation.cc:299] Negotiation successful
0806 06:30:34.004867 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260806 06:30:34.009083 20307 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 06:30:34.009738 20307 catalog_manager.cc:1504] Loading token signing keys...
I20260806 06:30:34.016299 20307 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c69c0d74ed1144c199deb95be6c914b5: Generated new TSK 0
I20260806 06:30:34.016479 20307 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 06:30:34.019752 21554 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHhLhT/build/release/bin/kudu with pid 20188
W20260806 06:30:34.031384 20166 connection.cc:535] client connection to 127.0.0.1:46391 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46391): Connection reset by peer (error 104)
2026-08-06T06:30:34Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1385 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785997581049705-21554-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 06:30:34 dist-test-slave-4tdj krb5kdc[20324](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 06:30:34 dist-test-slave-4tdj krb5kdc[20324](info): set up 2 sockets
Aug 06 06:30:34 dist-test-slave-4tdj krb5kdc[20324](info): commencing operation
krb5kdc: starting...
W20260806 06:30:35.164815 21554 mini_kdc.cc:121] Time spent starting KDC: real 1.120s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785997581049705-21554-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785997581049705-21554-0/krb5kdc/test-user.keytab.
Aug 06 06:30:35 dist-test-slave-4tdj krb5kdc[20324](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997835, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T06:30:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T06:30:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785997581049705-21554-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785997581049705-21554-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 06:30:35.283139 21554 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHhLhT/build/release/bin/kudu
/tmp/dist-test-taskFHhLhT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785997581049705-21554-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785997581049705-21554-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785997581049705-21554-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785997581049705-21554-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40837
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785997581049705-21554-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40837
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785997581049705-21554-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785997581049705-21554-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785997581049705-21554-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 06:30:35.363687 20340 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 06:30:35.363888 20340 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 06:30:35.363911 20340 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 06:30:35.365033 20340 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 06:30:35.365069 20340 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 06:30:35.365082 20340 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 06:30:35.365092 20340 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 06:30:35.365103 20340 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 06:30:35.366603 20340 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785997581049705-21554-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785997581049705-21554-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40837
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785997581049705-21554-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40837
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785997581049705-21554-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20340
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785997581049705-21554-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 05:11:02 UTC on ccycloud.quasar-hvzbbz.root.comops.site
build id 4569617
I20260806 06:30:35.366806 20340 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 06:30:35.366989 20340 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 06:30:35.371781 20351 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:30:35.371774 20348 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:30:35.371884 20349 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:30:35.372012 20340 server_base.cc:1034] running on GCE node
I20260806 06:30:35.372273 20340 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 06:30:35.372551 20340 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 06:30:35.373687 20340 hybrid_clock.cc:648] HybridClock initialized: now 1785997835373674 us; error 30 us; skew 500 ppm
Aug 06 06:30:35 dist-test-slave-4tdj krb5kdc[20324](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997835, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 06:30:35.376103 20340 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 06:30:35.376546 20340 webserver.cc:460] Webserver started at http://127.0.0.1:37255/ using document root <none> and password file <none>
I20260806 06:30:35.376773 20340 fs_manager.cc:359] Metadata directory not provided
I20260806 06:30:35.376819 20340 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 06:30:35.376941 20340 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 06:30:35.377971 20340 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785997581049705-21554-0/minicluster-data/master-0/data/instance:
uuid: "967bf5722574435d86f611f445fb0f1f"
format_stamp: "Formatted at 2026-08-06 06:30:35 on dist-test-slave-4tdj"
server_key: "edd0d5601c9ce4bfe8bf80590a592e16"
server_key_iv: "bdddbafdbe27bea3b7ebe08944e966bc"
server_key_version: "encryptionkey@0"
I20260806 06:30:35.378317 20340 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785997581049705-21554-0/minicluster-data/master-0/wal/instance:
uuid: "967bf5722574435d86f611f445fb0f1f"
format_stamp: "Formatted at 2026-08-06 06:30:35 on dist-test-slave-4tdj"
server_key: "edd0d5601c9ce4bfe8bf80590a592e16"
server_key_iv: "bdddbafdbe27bea3b7ebe08944e966bc"
server_key_version: "encryptionkey@0"
I20260806 06:30:35.379594 20340 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260806 06:30:35.380314 20358 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 06:30:35.380472 20340 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260806 06:30:35.380569 20340 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785997581049705-21554-0/minicluster-data/master-0/data,/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785997581049705-21554-0/minicluster-data/master-0/wal
uuid: "967bf5722574435d86f611f445fb0f1f"
format_stamp: "Formatted at 2026-08-06 06:30:35 on dist-test-slave-4tdj"
server_key: "edd0d5601c9ce4bfe8bf80590a592e16"
server_key_iv: "bdddbafdbe27bea3b7ebe08944e966bc"
server_key_version: "encryptionkey@0"
I20260806 06:30:35.380631 20340 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785997581049705-21554-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785997581049705-21554-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785997581049705-21554-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 06:30:35.383919 20340 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 06:30:35.385993 20340 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 06:30:35.386124 20340 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 06:30:35.389911 20340 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40837
I20260806 06:30:35.389952 20434 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40837 every 8 connection(s)
I20260806 06:30:35.390273 20340 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785997581049705-21554-0/minicluster-data/master-0/data/info.pb
I20260806 06:30:35.391266 20435 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 06:30:35.393502 20435 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 967bf5722574435d86f611f445fb0f1f: Bootstrap starting.
I20260806 06:30:35.394099 20435 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 967bf5722574435d86f611f445fb0f1f: Neither blocks nor log segments found. Creating new log.
I20260806 06:30:35.394457 20435 log.cc:824] T 00000000000000000000000000000000 P 967bf5722574435d86f611f445fb0f1f: Log is configured to *not* fsync() on all Append() calls
I20260806 06:30:35.395174 20435 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 967bf5722574435d86f611f445fb0f1f: No bootstrap required, opened a new log
I20260806 06:30:35.396780 20435 raft_consensus.cc:348] T 00000000000000000000000000000000 P 967bf5722574435d86f611f445fb0f1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "967bf5722574435d86f611f445fb0f1f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40837 } }
I20260806 06:30:35.396920 20435 raft_consensus.cc:374] T 00000000000000000000000000000000 P 967bf5722574435d86f611f445fb0f1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 06:30:35.396952 20435 raft_consensus.cc:729] T 00000000000000000000000000000000 P 967bf5722574435d86f611f445fb0f1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 967bf5722574435d86f611f445fb0f1f, State: Initialized, Role: FOLLOWER
I20260806 06:30:35.397094 20435 consensus_queue.cc:260] T 00000000000000000000000000000000 P 967bf5722574435d86f611f445fb0f1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "967bf5722574435d86f611f445fb0f1f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40837 } }
I20260806 06:30:35.397154 20435 raft_consensus.cc:388] T 00000000000000000000000000000000 P 967bf5722574435d86f611f445fb0f1f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 06:30:35.397176 20435 raft_consensus.cc:482] T 00000000000000000000000000000000 P 967bf5722574435d86f611f445fb0f1f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 06:30:35.397219 20435 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 967bf5722574435d86f611f445fb0f1f [term 0 FOLLOWER]: Advancing to term 1
I20260806 06:30:35.397764 20435 raft_consensus.cc:504] T 00000000000000000000000000000000 P 967bf5722574435d86f611f445fb0f1f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "967bf5722574435d86f611f445fb0f1f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40837 } }
I20260806 06:30:35.397881 20435 leader_election.cc:302] T 00000000000000000000000000000000 P 967bf5722574435d86f611f445fb0f1f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 967bf5722574435d86f611f445fb0f1f; no voters:
I20260806 06:30:35.397904 21554 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHhLhT/build/release/bin/kudu as pid 20340
I20260806 06:30:35.398002 21554 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785997581049705-21554-0/minicluster-data/master-0/wal/instance
I20260806 06:30:35.398057 20435 leader_election.cc:288] T 00000000000000000000000000000000 P 967bf5722574435d86f611f445fb0f1f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 06:30:35.398100 20439 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 967bf5722574435d86f611f445fb0f1f [term 1 FOLLOWER]: Leader election won for term 1
I20260806 06:30:35.398207 21554 external_mini_cluster.cc:1428] Setting key c7faff4a36b6ce95c295aa732073043c
I20260806 06:30:35.398244 20439 raft_consensus.cc:686] T 00000000000000000000000000000000 P 967bf5722574435d86f611f445fb0f1f [term 1 LEADER]: Becoming Leader. State: Replica: 967bf5722574435d86f611f445fb0f1f, State: Running, Role: LEADER
I20260806 06:30:35.398363 20435 sys_catalog.cc:564] T 00000000000000000000000000000000 P 967bf5722574435d86f611f445fb0f1f [sys.catalog]: configured and running, proceeding with master startup.
I20260806 06:30:35.398404 20439 consensus_queue.cc:237] T 00000000000000000000000000000000 P 967bf5722574435d86f611f445fb0f1f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "967bf5722574435d86f611f445fb0f1f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40837 } }
I20260806 06:30:35.398749 20439 sys_catalog.cc:455] T 00000000000000000000000000000000 P 967bf5722574435d86f611f445fb0f1f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 967bf5722574435d86f611f445fb0f1f. Latest consensus state: current_term: 1 leader_uuid: "967bf5722574435d86f611f445fb0f1f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "967bf5722574435d86f611f445fb0f1f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40837 } } }
I20260806 06:30:35.398861 20443 sys_catalog.cc:455] T 00000000000000000000000000000000 P 967bf5722574435d86f611f445fb0f1f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "967bf5722574435d86f611f445fb0f1f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "967bf5722574435d86f611f445fb0f1f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40837 } } }
I20260806 06:30:35.399080 20439 sys_catalog.cc:458] T 00000000000000000000000000000000 P 967bf5722574435d86f611f445fb0f1f [sys.catalog]: This master's current role is: LEADER
I20260806 06:30:35.399085 20443 sys_catalog.cc:458] T 00000000000000000000000000000000 P 967bf5722574435d86f611f445fb0f1f [sys.catalog]: This master's current role is: LEADER
I20260806 06:30:35.399309 20456 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 06:30:35.399904 20456 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 06:30:35.401063 20454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:35.398881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.56:36171 (local address 127.0.0.1:40837)
0806 06:30:35.399274 (+ 393us) server_negotiation.cc:206] Beginning negotiation
0806 06:30:35.399284 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 06:30:35.399318 (+ 34us) server_negotiation.cc:407] Connection header received
0806 06:30:35.399616 (+ 298us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:30:35.399626 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:30:35.399687 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:30:35.400129 (+ 442us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:30:35.400599 (+ 470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:30:35.400602 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:30:35.400602 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 06:30:35.400605 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:30:35.400652 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 06:30:35.400678 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:30:35.400738 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0806 06:30:35.400851 (+ 113us) server_negotiation.cc:299] Negotiation successful
0806 06:30:35.400896 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":82,"threads_started":1}
I20260806 06:30:35.401666 20456 catalog_manager.cc:1347] Generated new cluster ID: 8b2f59de57f64e2385bbf3106c0b55e6
I20260806 06:30:35.401711 20456 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 06:30:35.402071 21554 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 06:30:35.404590 20454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:35.403518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.56:55647 (local address 127.0.0.1:40837)
0806 06:30:35.403555 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0806 06:30:35.403557 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:30:35.403671 (+ 114us) server_negotiation.cc:407] Connection header received
0806 06:30:35.403752 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:30:35.403753 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:30:35.403758 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:30:35.403781 (+ 23us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:30:35.404391 (+ 610us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:30:35.404392 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:30:35.404392 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 06:30:35.404395 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:30:35.404409 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 06:30:35.404413 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:30:35.404437 (+ 24us) server_negotiation.cc:1035] Waiting for connection context
0806 06:30:35.404516 (+ 79us) server_negotiation.cc:299] Negotiation successful
0806 06:30:35.404529 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260806 06:30:35.424111 20456 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 06:30:35.424330 20456 catalog_manager.cc:1504] Loading token signing keys...
I20260806 06:30:35.429960 20456 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 967bf5722574435d86f611f445fb0f1f: Generated new TSK 0
I20260806 06:30:35.430176 20456 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 06:30:35.431849 21554 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHhLhT/build/release/bin/kudu with pid 20340
2026-08-06T06:30:35Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1414 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785997581049705-21554-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 06:30:35 dist-test-slave-4tdj krb5kdc[20480](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 06:30:35 dist-test-slave-4tdj krb5kdc[20480](info): set up 2 sockets
Aug 06 06:30:35 dist-test-slave-4tdj krb5kdc[20480](info): commencing operation
krb5kdc: starting...
W20260806 06:30:36.543292 21554 mini_kdc.cc:121] Time spent starting KDC: real 1.084s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785997581049705-21554-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785997581049705-21554-0/krb5kdc/test-user.keytab.
Aug 06 06:30:36 dist-test-slave-4tdj krb5kdc[20480](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997836, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T06:30:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T06:30:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785997581049705-21554-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785997581049705-21554-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 06:30:36.654915 21554 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHhLhT/build/release/bin/kudu
/tmp/dist-test-taskFHhLhT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785997581049705-21554-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785997581049705-21554-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785997581049705-21554-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785997581049705-21554-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36245
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785997581049705-21554-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36245
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785997581049705-21554-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785997581049705-21554-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785997581049705-21554-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 06:30:36.736135 20500 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 06:30:36.736346 20500 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 06:30:36.736377 20500 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 06:30:36.737494 20500 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 06:30:36.737540 20500 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 06:30:36.737561 20500 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 06:30:36.737581 20500 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 06:30:36.737598 20500 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 06:30:36.739063 20500 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785997581049705-21554-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785997581049705-21554-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36245
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785997581049705-21554-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36245
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785997581049705-21554-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20500
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785997581049705-21554-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 05:11:02 UTC on ccycloud.quasar-hvzbbz.root.comops.site
build id 4569617
I20260806 06:30:36.739310 20500 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 06:30:36.739528 20500 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 06:30:36.744218 20509 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:30:36.744259 20508 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:30:36.744392 20500 server_base.cc:1034] running on GCE node
W20260806 06:30:36.744232 20511 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:30:36.744690 20500 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 06:30:36.744940 20500 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 06:30:36.746121 20500 hybrid_clock.cc:648] HybridClock initialized: now 1785997836746103 us; error 33 us; skew 500 ppm
Aug 06 06:30:36 dist-test-slave-4tdj krb5kdc[20480](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997836, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 06:30:36.748613 20500 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 06:30:36.749096 20500 webserver.cc:460] Webserver started at http://127.0.0.1:39071/ using document root <none> and password file <none>
I20260806 06:30:36.749279 20500 fs_manager.cc:359] Metadata directory not provided
I20260806 06:30:36.749320 20500 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 06:30:36.749418 20500 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 06:30:36.750490 20500 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785997581049705-21554-0/minicluster-data/master-0/data/instance:
uuid: "9ade9ad66c9a43778451a3090679774b"
format_stamp: "Formatted at 2026-08-06 06:30:36 on dist-test-slave-4tdj"
server_key: "668c5115419fa8fb44b26415df265789"
server_key_iv: "0f4574ea3cda30f8954a0244fcf353f5"
server_key_version: "encryptionkey@0"
I20260806 06:30:36.750805 20500 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785997581049705-21554-0/minicluster-data/master-0/wal/instance:
uuid: "9ade9ad66c9a43778451a3090679774b"
format_stamp: "Formatted at 2026-08-06 06:30:36 on dist-test-slave-4tdj"
server_key: "668c5115419fa8fb44b26415df265789"
server_key_iv: "0f4574ea3cda30f8954a0244fcf353f5"
server_key_version: "encryptionkey@0"
I20260806 06:30:36.752049 20500 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260806 06:30:36.752737 20521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 06:30:36.752893 20500 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260806 06:30:36.752997 20500 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785997581049705-21554-0/minicluster-data/master-0/data,/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785997581049705-21554-0/minicluster-data/master-0/wal
uuid: "9ade9ad66c9a43778451a3090679774b"
format_stamp: "Formatted at 2026-08-06 06:30:36 on dist-test-slave-4tdj"
server_key: "668c5115419fa8fb44b26415df265789"
server_key_iv: "0f4574ea3cda30f8954a0244fcf353f5"
server_key_version: "encryptionkey@0"
I20260806 06:30:36.753063 20500 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785997581049705-21554-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785997581049705-21554-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785997581049705-21554-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 06:30:36.778689 20500 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 06:30:36.781045 20500 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 06:30:36.781200 20500 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 06:30:36.785405 20500 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36245
I20260806 06:30:36.785449 20591 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36245 every 8 connection(s)
I20260806 06:30:36.785802 20500 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785997581049705-21554-0/minicluster-data/master-0/data/info.pb
I20260806 06:30:36.786716 20592 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 06:30:36.788996 20592 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ade9ad66c9a43778451a3090679774b: Bootstrap starting.
I20260806 06:30:36.789137 21554 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHhLhT/build/release/bin/kudu as pid 20500
I20260806 06:30:36.789243 21554 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785997581049705-21554-0/minicluster-data/master-0/wal/instance
I20260806 06:30:36.789464 21554 external_mini_cluster.cc:1428] Setting key 4ca67b3f6bb582d16e984e3ff50c7da3
I20260806 06:30:36.789712 20592 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9ade9ad66c9a43778451a3090679774b: Neither blocks nor log segments found. Creating new log.
I20260806 06:30:36.790012 20592 log.cc:824] T 00000000000000000000000000000000 P 9ade9ad66c9a43778451a3090679774b: Log is configured to *not* fsync() on all Append() calls
I20260806 06:30:36.790748 20592 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ade9ad66c9a43778451a3090679774b: No bootstrap required, opened a new log
I20260806 06:30:36.792877 20592 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9ade9ad66c9a43778451a3090679774b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ade9ad66c9a43778451a3090679774b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36245 } }
I20260806 06:30:36.793051 20592 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9ade9ad66c9a43778451a3090679774b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 06:30:36.793085 20592 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9ade9ad66c9a43778451a3090679774b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ade9ad66c9a43778451a3090679774b, State: Initialized, Role: FOLLOWER
I20260806 06:30:36.793217 20592 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9ade9ad66c9a43778451a3090679774b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ade9ad66c9a43778451a3090679774b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36245 } }
I20260806 06:30:36.793288 20592 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9ade9ad66c9a43778451a3090679774b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 06:30:36.793321 20592 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9ade9ad66c9a43778451a3090679774b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 06:30:36.793362 20592 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9ade9ad66c9a43778451a3090679774b [term 0 FOLLOWER]: Advancing to term 1
I20260806 06:30:36.793952 20592 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9ade9ad66c9a43778451a3090679774b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ade9ad66c9a43778451a3090679774b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36245 } }
I20260806 06:30:36.794054 20592 leader_election.cc:302] T 00000000000000000000000000000000 P 9ade9ad66c9a43778451a3090679774b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9ade9ad66c9a43778451a3090679774b; no voters:
I20260806 06:30:36.794260 20592 leader_election.cc:288] T 00000000000000000000000000000000 P 9ade9ad66c9a43778451a3090679774b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 06:30:36.794329 20599 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9ade9ad66c9a43778451a3090679774b [term 1 FOLLOWER]: Leader election won for term 1
I20260806 06:30:36.794545 20592 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9ade9ad66c9a43778451a3090679774b [sys.catalog]: configured and running, proceeding with master startup.
I20260806 06:30:36.794761 20599 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9ade9ad66c9a43778451a3090679774b [term 1 LEADER]: Becoming Leader. State: Replica: 9ade9ad66c9a43778451a3090679774b, State: Running, Role: LEADER
I20260806 06:30:36.794864 20599 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9ade9ad66c9a43778451a3090679774b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ade9ad66c9a43778451a3090679774b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36245 } }
I20260806 06:30:36.795207 20599 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ade9ad66c9a43778451a3090679774b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9ade9ad66c9a43778451a3090679774b. Latest consensus state: current_term: 1 leader_uuid: "9ade9ad66c9a43778451a3090679774b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ade9ad66c9a43778451a3090679774b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36245 } } }
I20260806 06:30:36.795217 20601 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ade9ad66c9a43778451a3090679774b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9ade9ad66c9a43778451a3090679774b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ade9ad66c9a43778451a3090679774b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36245 } } }
I20260806 06:30:36.795331 20599 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ade9ad66c9a43778451a3090679774b [sys.catalog]: This master's current role is: LEADER
I20260806 06:30:36.795351 20601 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ade9ad66c9a43778451a3090679774b [sys.catalog]: This master's current role is: LEADER
I20260806 06:30:36.795521 20610 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 06:30:36.796022 20610 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Aug 06 06:30:36 dist-test-slave-4tdj krb5kdc[20480](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997836, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260806 06:30:36.797904 20610 catalog_manager.cc:1347] Generated new cluster ID: a264dd70488045df9d52f65a356fcbae
I20260806 06:30:36.797961 20610 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 06:30:36.800055 20596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:36.789998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.56:45155 (local address 127.0.0.1:36245)
0806 06:30:36.790222 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0806 06:30:36.790233 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 06:30:36.790269 (+ 36us) server_negotiation.cc:407] Connection header received
0806 06:30:36.790790 (+ 521us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:30:36.790797 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:30:36.790873 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:30:36.791005 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:30:36.791753 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:36.792632 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:36.793623 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:36.794025 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:36.794123 (+ 98us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:30:36.796925 (+ 2802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:30:36.796929 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:30:36.796930 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 06:30:36.796942 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:30:36.798641 (+ 1699us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:30:36.799001 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:30:36.799005 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:30:36.799005 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:30:36.799093 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:30:36.799290 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:30:36.799291 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:30:36.799291 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:30:36.799525 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:30:36.799690 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0806 06:30:36.799785 (+ 95us) server_negotiation.cc:299] Negotiation successful
0806 06:30:36.799883 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":62,"threads_started":1}
I20260806 06:30:36.801019 21554 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 06:30:36.811937 20596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:36.802479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.56:46479 (local address 127.0.0.1:36245)
0806 06:30:36.802519 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0806 06:30:36.802521 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:30:36.806464 (+ 3943us) server_negotiation.cc:407] Connection header received
0806 06:30:36.806528 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:30:36.806529 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:30:36.806539 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:30:36.806585 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:30:36.807111 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:36.807624 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:36.808289 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:36.808503 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:36.808556 (+ 53us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:30:36.809645 (+ 1089us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:30:36.809647 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:30:36.809647 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 06:30:36.809654 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:30:36.810897 (+ 1243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:30:36.811245 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:30:36.811246 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:30:36.811246 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:30:36.811298 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:30:36.811489 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:30:36.811490 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:30:36.811490 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:30:36.811591 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:30:36.811678 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0806 06:30:36.811792 (+ 114us) server_negotiation.cc:299] Negotiation successful
0806 06:30:36.811849 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260806 06:30:36.815541 20610 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 06:30:36.816213 20610 catalog_manager.cc:1504] Loading token signing keys...
I20260806 06:30:36.822993 20610 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9ade9ad66c9a43778451a3090679774b: Generated new TSK 0
I20260806 06:30:36.823181 20610 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 06:30:36.833130 21554 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHhLhT/build/release/bin/kudu with pid 20500
W20260806 06:30:36.844363 20478 connection.cc:535] client connection to 127.0.0.1:36245 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36245): Connection reset by peer (error 104)
2026-08-06T06:30:36Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1398 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785997581049705-21554-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 06:30:36 dist-test-slave-4tdj krb5kdc[20631](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 06:30:36 dist-test-slave-4tdj krb5kdc[20631](info): set up 2 sockets
Aug 06 06:30:36 dist-test-slave-4tdj krb5kdc[20631](info): commencing operation
krb5kdc: starting...
W20260806 06:30:37.957233 21554 mini_kdc.cc:121] Time spent starting KDC: real 1.099s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785997581049705-21554-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785997581049705-21554-0/krb5kdc/test-user.keytab.
Aug 06 06:30:38 dist-test-slave-4tdj krb5kdc[20631](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997838, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T06:30:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T06:30:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785997581049705-21554-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785997581049705-21554-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 06:30:38.072783 21554 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHhLhT/build/release/bin/kudu
/tmp/dist-test-taskFHhLhT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785997581049705-21554-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785997581049705-21554-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785997581049705-21554-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785997581049705-21554-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45221
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785997581049705-21554-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45221
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785997581049705-21554-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785997581049705-21554-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785997581049705-21554-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 06:30:38.152283 20650 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 06:30:38.152490 20650 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 06:30:38.152521 20650 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 06:30:38.153673 20650 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 06:30:38.153715 20650 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 06:30:38.153734 20650 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 06:30:38.153750 20650 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 06:30:38.153764 20650 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 06:30:38.155259 20650 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785997581049705-21554-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785997581049705-21554-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45221
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785997581049705-21554-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45221
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785997581049705-21554-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20650
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785997581049705-21554-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 05:11:02 UTC on ccycloud.quasar-hvzbbz.root.comops.site
build id 4569617
I20260806 06:30:38.155500 20650 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 06:30:38.155699 20650 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 06:30:38.160367 20661 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:30:38.160352 20657 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:30:38.160332 20659 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:30:38.160616 20650 server_base.cc:1034] running on GCE node
I20260806 06:30:38.160845 20650 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 06:30:38.161114 20650 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 06:30:38.162269 20650 hybrid_clock.cc:648] HybridClock initialized: now 1785997838162253 us; error 28 us; skew 500 ppm
Aug 06 06:30:38 dist-test-slave-4tdj krb5kdc[20631](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997838, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 06:30:38.167668 20650 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 06:30:38.168097 20650 webserver.cc:460] Webserver started at http://127.0.0.1:36447/ using document root <none> and password file <none>
I20260806 06:30:38.168295 20650 fs_manager.cc:359] Metadata directory not provided
I20260806 06:30:38.168334 20650 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 06:30:38.168423 20650 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 06:30:38.169409 20650 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785997581049705-21554-0/minicluster-data/master-0/data/instance:
uuid: "63d495c31d9d4002bee6ef6045ba2d91"
format_stamp: "Formatted at 2026-08-06 06:30:38 on dist-test-slave-4tdj"
server_key: "81333941fa78dddfbf8abe65799efd57"
server_key_iv: "0a9e324ee283a5c5d58a6a05f1d05809"
server_key_version: "encryptionkey@0"
I20260806 06:30:38.169704 20650 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785997581049705-21554-0/minicluster-data/master-0/wal/instance:
uuid: "63d495c31d9d4002bee6ef6045ba2d91"
format_stamp: "Formatted at 2026-08-06 06:30:38 on dist-test-slave-4tdj"
server_key: "81333941fa78dddfbf8abe65799efd57"
server_key_iv: "0a9e324ee283a5c5d58a6a05f1d05809"
server_key_version: "encryptionkey@0"
I20260806 06:30:38.170936 20650 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260806 06:30:38.171607 20672 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 06:30:38.171772 20650 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260806 06:30:38.171846 20650 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785997581049705-21554-0/minicluster-data/master-0/data,/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785997581049705-21554-0/minicluster-data/master-0/wal
uuid: "63d495c31d9d4002bee6ef6045ba2d91"
format_stamp: "Formatted at 2026-08-06 06:30:38 on dist-test-slave-4tdj"
server_key: "81333941fa78dddfbf8abe65799efd57"
server_key_iv: "0a9e324ee283a5c5d58a6a05f1d05809"
server_key_version: "encryptionkey@0"
I20260806 06:30:38.171901 20650 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785997581049705-21554-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785997581049705-21554-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785997581049705-21554-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 06:30:38.193794 20650 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 06:30:38.195668 20650 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 06:30:38.195786 20650 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 06:30:38.199549 20650 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45221
I20260806 06:30:38.199595 20744 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45221 every 8 connection(s)
I20260806 06:30:38.199913 20650 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785997581049705-21554-0/minicluster-data/master-0/data/info.pb
I20260806 06:30:38.200791 20746 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 06:30:38.203090 20746 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 63d495c31d9d4002bee6ef6045ba2d91: Bootstrap starting.
I20260806 06:30:38.203663 20746 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 63d495c31d9d4002bee6ef6045ba2d91: Neither blocks nor log segments found. Creating new log.
I20260806 06:30:38.203917 20746 log.cc:824] T 00000000000000000000000000000000 P 63d495c31d9d4002bee6ef6045ba2d91: Log is configured to *not* fsync() on all Append() calls
I20260806 06:30:38.204563 20746 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 63d495c31d9d4002bee6ef6045ba2d91: No bootstrap required, opened a new log
I20260806 06:30:38.206142 20746 raft_consensus.cc:348] T 00000000000000000000000000000000 P 63d495c31d9d4002bee6ef6045ba2d91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63d495c31d9d4002bee6ef6045ba2d91" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45221 } }
I20260806 06:30:38.206282 20746 raft_consensus.cc:374] T 00000000000000000000000000000000 P 63d495c31d9d4002bee6ef6045ba2d91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 06:30:38.206355 20746 raft_consensus.cc:729] T 00000000000000000000000000000000 P 63d495c31d9d4002bee6ef6045ba2d91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63d495c31d9d4002bee6ef6045ba2d91, State: Initialized, Role: FOLLOWER
I20260806 06:30:38.206499 20746 consensus_queue.cc:260] T 00000000000000000000000000000000 P 63d495c31d9d4002bee6ef6045ba2d91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63d495c31d9d4002bee6ef6045ba2d91" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45221 } }
I20260806 06:30:38.206589 20746 raft_consensus.cc:388] T 00000000000000000000000000000000 P 63d495c31d9d4002bee6ef6045ba2d91 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 06:30:38.206626 20746 raft_consensus.cc:482] T 00000000000000000000000000000000 P 63d495c31d9d4002bee6ef6045ba2d91 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 06:30:38.206674 20746 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 63d495c31d9d4002bee6ef6045ba2d91 [term 0 FOLLOWER]: Advancing to term 1
I20260806 06:30:38.206921 21554 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHhLhT/build/release/bin/kudu as pid 20650
I20260806 06:30:38.207026 21554 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785997581049705-21554-0/minicluster-data/master-0/wal/instance
I20260806 06:30:38.207214 21554 external_mini_cluster.cc:1428] Setting key ab19136bd052f7f595a0944f53b4d77d
I20260806 06:30:38.207259 20746 raft_consensus.cc:504] T 00000000000000000000000000000000 P 63d495c31d9d4002bee6ef6045ba2d91 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63d495c31d9d4002bee6ef6045ba2d91" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45221 } }
I20260806 06:30:38.207366 20746 leader_election.cc:302] T 00000000000000000000000000000000 P 63d495c31d9d4002bee6ef6045ba2d91 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 63d495c31d9d4002bee6ef6045ba2d91; no voters:
I20260806 06:30:38.207530 20746 leader_election.cc:288] T 00000000000000000000000000000000 P 63d495c31d9d4002bee6ef6045ba2d91 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 06:30:38.207580 20751 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 63d495c31d9d4002bee6ef6045ba2d91 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 06:30:38.207757 20751 raft_consensus.cc:686] T 00000000000000000000000000000000 P 63d495c31d9d4002bee6ef6045ba2d91 [term 1 LEADER]: Becoming Leader. State: Replica: 63d495c31d9d4002bee6ef6045ba2d91, State: Running, Role: LEADER
I20260806 06:30:38.207818 20746 sys_catalog.cc:564] T 00000000000000000000000000000000 P 63d495c31d9d4002bee6ef6045ba2d91 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 06:30:38.207885 20751 consensus_queue.cc:237] T 00000000000000000000000000000000 P 63d495c31d9d4002bee6ef6045ba2d91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63d495c31d9d4002bee6ef6045ba2d91" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45221 } }
I20260806 06:30:38.208243 20751 sys_catalog.cc:455] T 00000000000000000000000000000000 P 63d495c31d9d4002bee6ef6045ba2d91 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "63d495c31d9d4002bee6ef6045ba2d91" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63d495c31d9d4002bee6ef6045ba2d91" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45221 } } }
I20260806 06:30:38.208352 20751 sys_catalog.cc:458] T 00000000000000000000000000000000 P 63d495c31d9d4002bee6ef6045ba2d91 [sys.catalog]: This master's current role is: LEADER
I20260806 06:30:38.208520 20751 sys_catalog.cc:455] T 00000000000000000000000000000000 P 63d495c31d9d4002bee6ef6045ba2d91 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 63d495c31d9d4002bee6ef6045ba2d91. Latest consensus state: current_term: 1 leader_uuid: "63d495c31d9d4002bee6ef6045ba2d91" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63d495c31d9d4002bee6ef6045ba2d91" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45221 } } }
I20260806 06:30:38.208608 20751 sys_catalog.cc:458] T 00000000000000000000000000000000 P 63d495c31d9d4002bee6ef6045ba2d91 [sys.catalog]: This master's current role is: LEADER
W20260806 06:30:38.209740 20772 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 63d495c31d9d4002bee6ef6045ba2d91: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 06:30:38.210069 20772 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 06:30:38.210150 20765 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 06:30:38.210649 20765 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 06:30:38.212240 20765 catalog_manager.cc:1347] Generated new cluster ID: bd1318f66c104d96b67744f388766331
I20260806 06:30:38.212283 20765 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 06:30:38.215071 20756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:38.207639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.56:45013 (local address 127.0.0.1:45221)
0806 06:30:38.209907 (+ 2268us) server_negotiation.cc:206] Beginning negotiation
0806 06:30:38.209946 (+ 39us) server_negotiation.cc:399] Waiting for connection header
0806 06:30:38.209986 (+ 40us) server_negotiation.cc:407] Connection header received
0806 06:30:38.210183 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:30:38.210191 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:30:38.210256 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:30:38.210423 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:30:38.210728 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:38.211944 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:38.213013 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:38.213598 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:38.213696 (+ 98us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:30:38.214402 (+ 706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:30:38.214406 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:30:38.214406 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 06:30:38.214612 (+ 206us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:30:38.214647 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 06:30:38.214661 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:30:38.214735 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0806 06:30:38.214842 (+ 107us) server_negotiation.cc:299] Negotiation successful
0806 06:30:38.214867 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321,"thread_start_us":82,"threads_started":1}
I20260806 06:30:38.215945 21554 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 06:30:38.218268 20765 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 06:30:38.218961 20765 catalog_manager.cc:1504] Loading token signing keys...
I20260806 06:30:38.220890 20756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:38.217373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.56:48383 (local address 127.0.0.1:45221)
0806 06:30:38.217405 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0806 06:30:38.217407 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:30:38.218428 (+ 1021us) server_negotiation.cc:407] Connection header received
0806 06:30:38.218479 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:30:38.218480 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:30:38.218491 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:30:38.218538 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:30:38.218668 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:38.219135 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:38.219807 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:38.220020 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:38.220072 (+ 52us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:30:38.220675 (+ 603us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:30:38.220677 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:30:38.220677 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 06:30:38.220681 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:30:38.220695 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 06:30:38.220699 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:30:38.220743 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0806 06:30:38.220809 (+ 66us) server_negotiation.cc:299] Negotiation successful
0806 06:30:38.220820 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260806 06:30:38.223299 20765 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 63d495c31d9d4002bee6ef6045ba2d91: Generated new TSK 0
I20260806 06:30:38.223448 20765 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 06:30:38.227362 21554 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHhLhT/build/release/bin/kudu with pid 20650
W20260806 06:30:38.241495 20629 connection.cc:535] client connection to 127.0.0.1:45221 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45221): Connection reset by peer (error 104)
2026-08-06T06:30:38Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1397 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785997581049705-21554-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 06:30:38 dist-test-slave-4tdj krb5kdc[20787](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 06:30:38 dist-test-slave-4tdj krb5kdc[20787](info): set up 2 sockets
Aug 06 06:30:38 dist-test-slave-4tdj krb5kdc[20787](info): commencing operation
krb5kdc: starting...
W20260806 06:30:39.384439 21554 mini_kdc.cc:121] Time spent starting KDC: real 1.129s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785997581049705-21554-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785997581049705-21554-0/krb5kdc/test-user.keytab.
Aug 06 06:30:39 dist-test-slave-4tdj krb5kdc[20787](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997839, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T06:30:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T06:30:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785997581049705-21554-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785997581049705-21554-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 06:30:39.501477 21554 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHhLhT/build/release/bin/kudu
/tmp/dist-test-taskFHhLhT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785997581049705-21554-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785997581049705-21554-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785997581049705-21554-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785997581049705-21554-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44495
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785997581049705-21554-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44495
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785997581049705-21554-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785997581049705-21554-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785997581049705-21554-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 06:30:39.582792 20815 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 06:30:39.582989 20815 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 06:30:39.583011 20815 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 06:30:39.584103 20815 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 06:30:39.584141 20815 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 06:30:39.584153 20815 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 06:30:39.584164 20815 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 06:30:39.584179 20815 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 06:30:39.585616 20815 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785997581049705-21554-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785997581049705-21554-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44495
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785997581049705-21554-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44495
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785997581049705-21554-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20815
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785997581049705-21554-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 05:11:02 UTC on ccycloud.quasar-hvzbbz.root.comops.site
build id 4569617
I20260806 06:30:39.585800 20815 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 06:30:39.585980 20815 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 06:30:39.590569 20824 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:30:39.590586 20823 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:30:39.590677 20815 server_base.cc:1034] running on GCE node
W20260806 06:30:39.590572 20829 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:30:39.591076 20815 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 06:30:39.591362 20815 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 06:30:39.592504 20815 hybrid_clock.cc:648] HybridClock initialized: now 1785997839592493 us; error 29 us; skew 500 ppm
Aug 06 06:30:39 dist-test-slave-4tdj krb5kdc[20787](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997839, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 06:30:39.594909 20815 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 06:30:39.595289 20815 webserver.cc:460] Webserver started at http://127.0.0.1:38429/ using document root <none> and password file <none>
I20260806 06:30:39.595484 20815 fs_manager.cc:359] Metadata directory not provided
I20260806 06:30:39.595523 20815 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 06:30:39.595626 20815 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 06:30:39.596608 20815 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785997581049705-21554-0/minicluster-data/master-0/data/instance:
uuid: "3a7386ec1db0454eae70136b613f13e6"
format_stamp: "Formatted at 2026-08-06 06:30:39 on dist-test-slave-4tdj"
server_key: "ad4eba35d1e3d5f1f80db2cfceda1ecb"
server_key_iv: "d9c0ef4d63cbc413d779be73daefcd1b"
server_key_version: "encryptionkey@0"
I20260806 06:30:39.596900 20815 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785997581049705-21554-0/minicluster-data/master-0/wal/instance:
uuid: "3a7386ec1db0454eae70136b613f13e6"
format_stamp: "Formatted at 2026-08-06 06:30:39 on dist-test-slave-4tdj"
server_key: "ad4eba35d1e3d5f1f80db2cfceda1ecb"
server_key_iv: "d9c0ef4d63cbc413d779be73daefcd1b"
server_key_version: "encryptionkey@0"
I20260806 06:30:39.598063 20815 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260806 06:30:39.598773 20837 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 06:30:39.598909 20815 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260806 06:30:39.598994 20815 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785997581049705-21554-0/minicluster-data/master-0/data,/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785997581049705-21554-0/minicluster-data/master-0/wal
uuid: "3a7386ec1db0454eae70136b613f13e6"
format_stamp: "Formatted at 2026-08-06 06:30:39 on dist-test-slave-4tdj"
server_key: "ad4eba35d1e3d5f1f80db2cfceda1ecb"
server_key_iv: "d9c0ef4d63cbc413d779be73daefcd1b"
server_key_version: "encryptionkey@0"
I20260806 06:30:39.599058 20815 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785997581049705-21554-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785997581049705-21554-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785997581049705-21554-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 06:30:39.602239 20815 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 06:30:39.603932 20815 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 06:30:39.604053 20815 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 06:30:39.607841 20815 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44495
I20260806 06:30:39.607869 20904 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44495 every 8 connection(s)
I20260806 06:30:39.608166 20815 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785997581049705-21554-0/minicluster-data/master-0/data/info.pb
I20260806 06:30:39.609117 20905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 06:30:39.611368 20905 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a7386ec1db0454eae70136b613f13e6: Bootstrap starting.
I20260806 06:30:39.611927 20905 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3a7386ec1db0454eae70136b613f13e6: Neither blocks nor log segments found. Creating new log.
I20260806 06:30:39.612175 20905 log.cc:824] T 00000000000000000000000000000000 P 3a7386ec1db0454eae70136b613f13e6: Log is configured to *not* fsync() on all Append() calls
I20260806 06:30:39.612809 20905 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a7386ec1db0454eae70136b613f13e6: No bootstrap required, opened a new log
I20260806 06:30:39.614377 20905 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3a7386ec1db0454eae70136b613f13e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a7386ec1db0454eae70136b613f13e6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44495 } }
I20260806 06:30:39.614512 20905 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3a7386ec1db0454eae70136b613f13e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 06:30:39.614545 20905 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3a7386ec1db0454eae70136b613f13e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a7386ec1db0454eae70136b613f13e6, State: Initialized, Role: FOLLOWER
I20260806 06:30:39.614687 20905 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3a7386ec1db0454eae70136b613f13e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a7386ec1db0454eae70136b613f13e6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44495 } }
I20260806 06:30:39.614749 20905 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3a7386ec1db0454eae70136b613f13e6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 06:30:39.614773 20905 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3a7386ec1db0454eae70136b613f13e6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 06:30:39.614815 20905 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3a7386ec1db0454eae70136b613f13e6 [term 0 FOLLOWER]: Advancing to term 1
I20260806 06:30:39.615305 20905 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3a7386ec1db0454eae70136b613f13e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a7386ec1db0454eae70136b613f13e6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44495 } }
I20260806 06:30:39.615396 20905 leader_election.cc:302] T 00000000000000000000000000000000 P 3a7386ec1db0454eae70136b613f13e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3a7386ec1db0454eae70136b613f13e6; no voters:
I20260806 06:30:39.615543 20905 leader_election.cc:288] T 00000000000000000000000000000000 P 3a7386ec1db0454eae70136b613f13e6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 06:30:39.615576 20914 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3a7386ec1db0454eae70136b613f13e6 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 06:30:39.615700 20914 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3a7386ec1db0454eae70136b613f13e6 [term 1 LEADER]: Becoming Leader. State: Replica: 3a7386ec1db0454eae70136b613f13e6, State: Running, Role: LEADER
I20260806 06:30:39.615795 20905 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3a7386ec1db0454eae70136b613f13e6 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 06:30:39.615819 20914 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3a7386ec1db0454eae70136b613f13e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a7386ec1db0454eae70136b613f13e6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44495 } }
I20260806 06:30:39.616247 20916 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a7386ec1db0454eae70136b613f13e6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3a7386ec1db0454eae70136b613f13e6. Latest consensus state: current_term: 1 leader_uuid: "3a7386ec1db0454eae70136b613f13e6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a7386ec1db0454eae70136b613f13e6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44495 } } }
I20260806 06:30:39.616367 20915 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a7386ec1db0454eae70136b613f13e6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3a7386ec1db0454eae70136b613f13e6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a7386ec1db0454eae70136b613f13e6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44495 } } }
I20260806 06:30:39.616387 20916 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a7386ec1db0454eae70136b613f13e6 [sys.catalog]: This master's current role is: LEADER
I20260806 06:30:39.616458 20915 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a7386ec1db0454eae70136b613f13e6 [sys.catalog]: This master's current role is: LEADER
I20260806 06:30:39.616457 21554 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHhLhT/build/release/bin/kudu as pid 20815
I20260806 06:30:39.616564 21554 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785997581049705-21554-0/minicluster-data/master-0/wal/instance
I20260806 06:30:39.616770 21554 external_mini_cluster.cc:1428] Setting key 8764901ffbc9ffdbd22798e5e4f034e1
I20260806 06:30:39.616820 20925 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 06:30:39.617338 20925 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 06:30:39.618968 20925 catalog_manager.cc:1347] Generated new cluster ID: bf4a30a440b645098db3fb96b2d1e17d
I20260806 06:30:39.619028 20925 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 06:30:39.619390 20937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:39.617413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.56:39505 (local address 127.0.0.1:44495)
0806 06:30:39.617802 (+ 389us) server_negotiation.cc:206] Beginning negotiation
0806 06:30:39.617811 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 06:30:39.617844 (+ 33us) server_negotiation.cc:407] Connection header received
0806 06:30:39.617958 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:30:39.617965 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:30:39.617997 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:30:39.618153 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:30:39.618814 (+ 661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:30:39.618817 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:30:39.618818 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 06:30:39.618822 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:30:39.618863 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 06:30:39.618878 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:30:39.619145 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0806 06:30:39.619176 (+ 31us) server_negotiation.cc:299] Negotiation successful
0806 06:30:39.619219 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":79,"threads_started":1}
I20260806 06:30:39.620169 21554 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 06:30:39.623380 20937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:39.621532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.56:52957 (local address 127.0.0.1:44495)
0806 06:30:39.621562 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0806 06:30:39.621563 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0806 06:30:39.622520 (+ 957us) server_negotiation.cc:407] Connection header received
0806 06:30:39.622570 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:30:39.622571 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:30:39.622577 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:30:39.622646 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:30:39.623179 (+ 533us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:30:39.623181 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:30:39.623181 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 06:30:39.623185 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:30:39.623201 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 06:30:39.623206 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:30:39.623266 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0806 06:30:39.623308 (+ 42us) server_negotiation.cc:299] Negotiation successful
0806 06:30:39.623322 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260806 06:30:39.627974 20925 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 06:30:39.628126 20925 catalog_manager.cc:1504] Loading token signing keys...
I20260806 06:30:39.633440 20925 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3a7386ec1db0454eae70136b613f13e6: Generated new TSK 0
I20260806 06:30:39.633605 20925 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 06:30:39.639719 21554 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHhLhT/build/release/bin/kudu with pid 20815
2026-08-06T06:30:39Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1410 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785997581049705-21554-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 06:30:39 dist-test-slave-4tdj krb5kdc[20951](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 06:30:39 dist-test-slave-4tdj krb5kdc[20951](info): set up 2 sockets
Aug 06 06:30:39 dist-test-slave-4tdj krb5kdc[20951](info): commencing operation
krb5kdc: starting...
W20260806 06:30:40.764977 21554 mini_kdc.cc:121] Time spent starting KDC: real 1.100s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785997581049705-21554-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785997581049705-21554-0/krb5kdc/test-user.keytab.
Aug 06 06:30:40 dist-test-slave-4tdj krb5kdc[20951](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997840, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T06:30:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T06:30:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.12.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.12.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.12.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.190.keytab.
Entry for principal kudu/127.21.12.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.190.keytab.
I20260806 06:30:40.877432 21554 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHhLhT/build/release/bin/kudu
/tmp/dist-test-taskFHhLhT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785997581049705-21554-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785997581049705-21554-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785997581049705-21554-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785997581049705-21554-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.12.190:43157
--webserver_interface=127.21.12.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.190.keytab
--principal=kudu/127.21.12.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.148:46489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.12.190:43157
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785997581049705-21554-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785997581049705-21554-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785997581049705-21554-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 06:30:40.956815 20969 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 06:30:40.957015 20969 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 06:30:40.957037 20969 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 06:30:40.958168 20969 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 06:30:40.958215 20969 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 06:30:40.958228 20969 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 06:30:40.958240 20969 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 06:30:40.958250 20969 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 06:30:40.959654 20969 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.148:46489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785997581049705-21554-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785997581049705-21554-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.12.190:43157
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.190.keytab
--principal=kudu/127.21.12.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.12.190:43157
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785997581049705-21554-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.12.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20969
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785997581049705-21554-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 05:11:02 UTC on ccycloud.quasar-hvzbbz.root.comops.site
build id 4569617
I20260806 06:30:40.959839 20969 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 06:30:40.960004 20969 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 06:30:40.964748 20978 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:30:40.964772 20969 server_base.cc:1034] running on GCE node
W20260806 06:30:40.964725 20980 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:30:40.964717 20983 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:30:40.965168 20969 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 06:30:40.965478 20969 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 06:30:40.966625 20969 hybrid_clock.cc:648] HybridClock initialized: now 1785997840966603 us; error 42 us; skew 500 ppm
Aug 06 06:30:40 dist-test-slave-4tdj krb5kdc[20951](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997840, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 06:30:40.968904 20969 init.cc:377] Logged in from keytab as kudu/127.21.12.190@KRBTEST.COM (short username kudu)
I20260806 06:30:40.969276 20969 webserver.cc:460] Webserver started at http://127.21.12.190:42689/ using document root <none> and password file <none>
I20260806 06:30:40.969441 20969 fs_manager.cc:359] Metadata directory not provided
I20260806 06:30:40.969478 20969 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 06:30:40.969566 20969 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 06:30:40.970608 20969 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785997581049705-21554-0/minicluster-data/master-0/data/instance:
uuid: "e86fdfc90cdd44d9970be48e033f7814"
format_stamp: "Formatted at 2026-08-06 06:30:40 on dist-test-slave-4tdj"
server_key: "eec2ffa3e56a0695fd272bbdfb46fe31"
server_key_iv: "b25621935c36744f5dc1b236a9c3644e"
server_key_version: "encryptionkey@0"
I20260806 06:30:40.970906 20969 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785997581049705-21554-0/minicluster-data/master-0/wal/instance:
uuid: "e86fdfc90cdd44d9970be48e033f7814"
format_stamp: "Formatted at 2026-08-06 06:30:40 on dist-test-slave-4tdj"
server_key: "eec2ffa3e56a0695fd272bbdfb46fe31"
server_key_iv: "b25621935c36744f5dc1b236a9c3644e"
server_key_version: "encryptionkey@0"
I20260806 06:30:40.972059 20969 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260806 06:30:40.972685 20990 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 06:30:40.972836 20969 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260806 06:30:40.972918 20969 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785997581049705-21554-0/minicluster-data/master-0/data,/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785997581049705-21554-0/minicluster-data/master-0/wal
uuid: "e86fdfc90cdd44d9970be48e033f7814"
format_stamp: "Formatted at 2026-08-06 06:30:40 on dist-test-slave-4tdj"
server_key: "eec2ffa3e56a0695fd272bbdfb46fe31"
server_key_iv: "b25621935c36744f5dc1b236a9c3644e"
server_key_version: "encryptionkey@0"
I20260806 06:30:40.972981 20969 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785997581049705-21554-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785997581049705-21554-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785997581049705-21554-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 06:30:40.989158 20969 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 06:30:40.991008 20969 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 06:30:40.991128 20969 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 06:30:40.994874 20969 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.190:43157
I20260806 06:30:40.994913 21053 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.190:43157 every 8 connection(s)
I20260806 06:30:40.995242 20969 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785997581049705-21554-0/minicluster-data/master-0/data/info.pb
I20260806 06:30:40.996064 21054 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 06:30:40.998237 21054 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e86fdfc90cdd44d9970be48e033f7814: Bootstrap starting.
I20260806 06:30:40.998819 21054 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e86fdfc90cdd44d9970be48e033f7814: Neither blocks nor log segments found. Creating new log.
I20260806 06:30:40.999046 21054 log.cc:824] T 00000000000000000000000000000000 P e86fdfc90cdd44d9970be48e033f7814: Log is configured to *not* fsync() on all Append() calls
I20260806 06:30:40.999677 21054 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e86fdfc90cdd44d9970be48e033f7814: No bootstrap required, opened a new log
I20260806 06:30:41.001511 21054 raft_consensus.cc:348] T 00000000000000000000000000000000 P e86fdfc90cdd44d9970be48e033f7814 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e86fdfc90cdd44d9970be48e033f7814" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 43157 } }
I20260806 06:30:41.001685 21054 raft_consensus.cc:374] T 00000000000000000000000000000000 P e86fdfc90cdd44d9970be48e033f7814 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 06:30:41.001739 21054 raft_consensus.cc:729] T 00000000000000000000000000000000 P e86fdfc90cdd44d9970be48e033f7814 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e86fdfc90cdd44d9970be48e033f7814, State: Initialized, Role: FOLLOWER
I20260806 06:30:41.001752 21554 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHhLhT/build/release/bin/kudu as pid 20969
I20260806 06:30:41.001853 21554 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785997581049705-21554-0/minicluster-data/master-0/wal/instance
I20260806 06:30:41.001888 21054 consensus_queue.cc:260] T 00000000000000000000000000000000 P e86fdfc90cdd44d9970be48e033f7814 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e86fdfc90cdd44d9970be48e033f7814" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 43157 } }
I20260806 06:30:41.001968 21054 raft_consensus.cc:388] T 00000000000000000000000000000000 P e86fdfc90cdd44d9970be48e033f7814 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 06:30:41.002004 21054 raft_consensus.cc:482] T 00000000000000000000000000000000 P e86fdfc90cdd44d9970be48e033f7814 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 06:30:41.002044 21554 external_mini_cluster.cc:1428] Setting key c4e8d589cf402cbfd70d0197d16cd41b
I20260806 06:30:41.002049 21054 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e86fdfc90cdd44d9970be48e033f7814 [term 0 FOLLOWER]: Advancing to term 1
I20260806 06:30:41.003063 21054 raft_consensus.cc:504] T 00000000000000000000000000000000 P e86fdfc90cdd44d9970be48e033f7814 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e86fdfc90cdd44d9970be48e033f7814" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 43157 } }
I20260806 06:30:41.003177 21054 leader_election.cc:302] T 00000000000000000000000000000000 P e86fdfc90cdd44d9970be48e033f7814 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e86fdfc90cdd44d9970be48e033f7814; no voters:
I20260806 06:30:41.003332 21054 leader_election.cc:288] T 00000000000000000000000000000000 P e86fdfc90cdd44d9970be48e033f7814 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 06:30:41.003357 21064 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e86fdfc90cdd44d9970be48e033f7814 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 06:30:41.003513 21064 raft_consensus.cc:686] T 00000000000000000000000000000000 P e86fdfc90cdd44d9970be48e033f7814 [term 1 LEADER]: Becoming Leader. State: Replica: e86fdfc90cdd44d9970be48e033f7814, State: Running, Role: LEADER
I20260806 06:30:41.003592 21054 sys_catalog.cc:564] T 00000000000000000000000000000000 P e86fdfc90cdd44d9970be48e033f7814 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 06:30:41.003604 21064 consensus_queue.cc:237] T 00000000000000000000000000000000 P e86fdfc90cdd44d9970be48e033f7814 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e86fdfc90cdd44d9970be48e033f7814" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 43157 } }
I20260806 06:30:41.004073 21066 sys_catalog.cc:455] T 00000000000000000000000000000000 P e86fdfc90cdd44d9970be48e033f7814 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e86fdfc90cdd44d9970be48e033f7814" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e86fdfc90cdd44d9970be48e033f7814" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 43157 } } }
I20260806 06:30:41.004201 21066 sys_catalog.cc:458] T 00000000000000000000000000000000 P e86fdfc90cdd44d9970be48e033f7814 [sys.catalog]: This master's current role is: LEADER
W20260806 06:30:41.005329 21087 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e86fdfc90cdd44d9970be48e033f7814: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 06:30:41.005430 21087 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 06:30:41.005578 21069 sys_catalog.cc:455] T 00000000000000000000000000000000 P e86fdfc90cdd44d9970be48e033f7814 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e86fdfc90cdd44d9970be48e033f7814. Latest consensus state: current_term: 1 leader_uuid: "e86fdfc90cdd44d9970be48e033f7814" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e86fdfc90cdd44d9970be48e033f7814" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 43157 } } }
I20260806 06:30:41.005661 21069 sys_catalog.cc:458] T 00000000000000000000000000000000 P e86fdfc90cdd44d9970be48e033f7814 [sys.catalog]: This master's current role is: LEADER
I20260806 06:30:41.005767 21079 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 06:30:41.005939 21079 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 06:30:41.008322 21079 catalog_manager.cc:1347] Generated new cluster ID: cc64ef0d161740c585e02e85d9371810
I20260806 06:30:41.008376 21079 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 06 06:30:41 dist-test-slave-4tdj krb5kdc[20951](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997840, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.12.190@KRBTEST.COM
I20260806 06:30:41.013548 21079 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 06:30:41.013976 21063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:41.002573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59445 (local address 127.21.12.190:43157)
0806 06:30:41.002739 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0806 06:30:41.002748 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 06:30:41.002775 (+ 27us) server_negotiation.cc:407] Connection header received
0806 06:30:41.002873 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:30:41.002878 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:30:41.002918 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:30:41.003020 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:30:41.003838 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:41.005319 (+ 1481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:41.006391 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:41.006878 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:41.006989 (+ 111us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:30:41.009427 (+ 2438us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:30:41.009431 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:30:41.009432 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 06:30:41.009444 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:30:41.011994 (+ 2550us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:30:41.012412 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:30:41.012415 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:30:41.012416 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:30:41.012535 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:30:41.012762 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:30:41.012763 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:30:41.012763 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:30:41.013097 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:30:41.013272 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0806 06:30:41.013720 (+ 448us) server_negotiation.cc:299] Negotiation successful
0806 06:30:41.013818 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":76,"threads_started":1}
I20260806 06:30:41.014233 21079 catalog_manager.cc:1504] Loading token signing keys...
I20260806 06:30:41.014920 21554 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 06:30:41.023306 21079 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e86fdfc90cdd44d9970be48e033f7814: Generated new TSK 0
I20260806 06:30:41.023523 21079 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 06:30:41.028203 21063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:41.016149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57869 (local address 127.21.12.190:43157)
0806 06:30:41.016183 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0806 06:30:41.016185 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:30:41.019543 (+ 3358us) server_negotiation.cc:407] Connection header received
0806 06:30:41.019690 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:30:41.019691 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:30:41.019727 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:30:41.019757 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:30:41.020804 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:41.021917 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:41.023040 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:41.023503 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:41.023600 (+ 97us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:30:41.025357 (+ 1757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:30:41.025359 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:30:41.025359 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 06:30:41.025367 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:30:41.026982 (+ 1615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:30:41.027379 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:30:41.027380 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:30:41.027380 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:30:41.027443 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:30:41.027704 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:30:41.027705 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:30:41.027705 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:30:41.027839 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:30:41.027942 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0806 06:30:41.028055 (+ 113us) server_negotiation.cc:299] Negotiation successful
0806 06:30:41.028121 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":16}
I20260806 06:30:41.030109 21554 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHhLhT/build/release/bin/kudu with pid 20969
W20260806 06:30:41.038401 20946 connection.cc:535] client connection to 127.21.12.190:43157 recv error: Network error: failed to read from TLS socket (remote: 127.21.12.190:43157): Connection reset by peer (error 104)
2026-08-06T06:30:41Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1386 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785997581049705-21554-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 06:30:41 dist-test-slave-4tdj krb5kdc[21098](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 06:30:41 dist-test-slave-4tdj krb5kdc[21098](info): set up 2 sockets
Aug 06 06:30:41 dist-test-slave-4tdj krb5kdc[21098](info): commencing operation
krb5kdc: starting...
W20260806 06:30:42.151813 21554 mini_kdc.cc:121] Time spent starting KDC: real 1.100s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785997581049705-21554-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785997581049705-21554-0/krb5kdc/test-user.keytab.
Aug 06 06:30:42 dist-test-slave-4tdj krb5kdc[21098](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997842, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T06:30:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T06:30:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.12.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.12.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.12.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.190.keytab.
Entry for principal kudu/127.21.12.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.190.keytab.
I20260806 06:30:42.284670 21554 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHhLhT/build/release/bin/kudu
/tmp/dist-test-taskFHhLhT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785997581049705-21554-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785997581049705-21554-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785997581049705-21554-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785997581049705-21554-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.12.190:45659
--webserver_interface=127.21.12.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.190.keytab
--principal=kudu/127.21.12.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.148:43993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.12.190:45659
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785997581049705-21554-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785997581049705-21554-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785997581049705-21554-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 06:30:42.363191 21115 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 06:30:42.363382 21115 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 06:30:42.363404 21115 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 06:30:42.364473 21115 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 06:30:42.364504 21115 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 06:30:42.364517 21115 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 06:30:42.364529 21115 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 06:30:42.364542 21115 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 06:30:42.365965 21115 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.148:43993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785997581049705-21554-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785997581049705-21554-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.12.190:45659
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.190.keytab
--principal=kudu/127.21.12.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.12.190:45659
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785997581049705-21554-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.12.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21115
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785997581049705-21554-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 05:11:02 UTC on ccycloud.quasar-hvzbbz.root.comops.site
build id 4569617
I20260806 06:30:42.366163 21115 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 06:30:42.366351 21115 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 06:30:42.370982 21120 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:30:42.371111 21115 server_base.cc:1034] running on GCE node
W20260806 06:30:42.371018 21123 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:30:42.370975 21121 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:30:42.371433 21115 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 06:30:42.371693 21115 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 06:30:42.372825 21115 hybrid_clock.cc:648] HybridClock initialized: now 1785997842372815 us; error 26 us; skew 500 ppm
Aug 06 06:30:42 dist-test-slave-4tdj krb5kdc[21098](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997842, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 06:30:42.375119 21115 init.cc:377] Logged in from keytab as kudu/127.21.12.190@KRBTEST.COM (short username kudu)
I20260806 06:30:42.375540 21115 webserver.cc:460] Webserver started at http://127.21.12.190:34479/ using document root <none> and password file <none>
I20260806 06:30:42.375788 21115 fs_manager.cc:359] Metadata directory not provided
I20260806 06:30:42.375836 21115 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 06:30:42.375959 21115 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 06:30:42.376929 21115 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785997581049705-21554-0/minicluster-data/master-0/data/instance:
uuid: "eb68676ca60d450fa5be3f864cf09dd3"
format_stamp: "Formatted at 2026-08-06 06:30:42 on dist-test-slave-4tdj"
server_key: "8cd094bc5fe78a81107d215220d40d41"
server_key_iv: "152e40d45d99de2c8bb7979c7b5b9d5c"
server_key_version: "encryptionkey@0"
I20260806 06:30:42.377249 21115 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785997581049705-21554-0/minicluster-data/master-0/wal/instance:
uuid: "eb68676ca60d450fa5be3f864cf09dd3"
format_stamp: "Formatted at 2026-08-06 06:30:42 on dist-test-slave-4tdj"
server_key: "8cd094bc5fe78a81107d215220d40d41"
server_key_iv: "152e40d45d99de2c8bb7979c7b5b9d5c"
server_key_version: "encryptionkey@0"
I20260806 06:30:42.378484 21115 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260806 06:30:42.379115 21133 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 06:30:42.379257 21115 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260806 06:30:42.379336 21115 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785997581049705-21554-0/minicluster-data/master-0/data,/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785997581049705-21554-0/minicluster-data/master-0/wal
uuid: "eb68676ca60d450fa5be3f864cf09dd3"
format_stamp: "Formatted at 2026-08-06 06:30:42 on dist-test-slave-4tdj"
server_key: "8cd094bc5fe78a81107d215220d40d41"
server_key_iv: "152e40d45d99de2c8bb7979c7b5b9d5c"
server_key_version: "encryptionkey@0"
I20260806 06:30:42.379396 21115 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785997581049705-21554-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785997581049705-21554-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785997581049705-21554-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 06:30:42.388336 21115 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 06:30:42.389918 21115 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 06:30:42.390034 21115 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 06:30:42.393747 21115 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.190:45659
I20260806 06:30:42.393783 21211 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.190:45659 every 8 connection(s)
I20260806 06:30:42.394083 21115 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785997581049705-21554-0/minicluster-data/master-0/data/info.pb
I20260806 06:30:42.394938 21212 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 06:30:42.397130 21212 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb68676ca60d450fa5be3f864cf09dd3: Bootstrap starting.
I20260806 06:30:42.397683 21212 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eb68676ca60d450fa5be3f864cf09dd3: Neither blocks nor log segments found. Creating new log.
I20260806 06:30:42.397933 21212 log.cc:824] T 00000000000000000000000000000000 P eb68676ca60d450fa5be3f864cf09dd3: Log is configured to *not* fsync() on all Append() calls
I20260806 06:30:42.398577 21212 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb68676ca60d450fa5be3f864cf09dd3: No bootstrap required, opened a new log
I20260806 06:30:42.398579 21554 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHhLhT/build/release/bin/kudu as pid 21115
I20260806 06:30:42.398689 21554 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785997581049705-21554-0/minicluster-data/master-0/wal/instance
I20260806 06:30:42.398874 21554 external_mini_cluster.cc:1428] Setting key a6fabe9675cda0ab3a570b780afe276b
I20260806 06:30:42.400424 21212 raft_consensus.cc:348] T 00000000000000000000000000000000 P eb68676ca60d450fa5be3f864cf09dd3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb68676ca60d450fa5be3f864cf09dd3" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 45659 } }
I20260806 06:30:42.400570 21212 raft_consensus.cc:374] T 00000000000000000000000000000000 P eb68676ca60d450fa5be3f864cf09dd3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 06:30:42.400606 21212 raft_consensus.cc:729] T 00000000000000000000000000000000 P eb68676ca60d450fa5be3f864cf09dd3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb68676ca60d450fa5be3f864cf09dd3, State: Initialized, Role: FOLLOWER
I20260806 06:30:42.400732 21212 consensus_queue.cc:260] T 00000000000000000000000000000000 P eb68676ca60d450fa5be3f864cf09dd3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb68676ca60d450fa5be3f864cf09dd3" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 45659 } }
I20260806 06:30:42.400812 21212 raft_consensus.cc:388] T 00000000000000000000000000000000 P eb68676ca60d450fa5be3f864cf09dd3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 06:30:42.400849 21212 raft_consensus.cc:482] T 00000000000000000000000000000000 P eb68676ca60d450fa5be3f864cf09dd3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 06:30:42.400897 21212 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eb68676ca60d450fa5be3f864cf09dd3 [term 0 FOLLOWER]: Advancing to term 1
I20260806 06:30:42.401400 21212 raft_consensus.cc:504] T 00000000000000000000000000000000 P eb68676ca60d450fa5be3f864cf09dd3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb68676ca60d450fa5be3f864cf09dd3" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 45659 } }
I20260806 06:30:42.401496 21212 leader_election.cc:302] T 00000000000000000000000000000000 P eb68676ca60d450fa5be3f864cf09dd3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eb68676ca60d450fa5be3f864cf09dd3; no voters:
I20260806 06:30:42.401644 21212 leader_election.cc:288] T 00000000000000000000000000000000 P eb68676ca60d450fa5be3f864cf09dd3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 06:30:42.401682 21219 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eb68676ca60d450fa5be3f864cf09dd3 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 06:30:42.401798 21219 raft_consensus.cc:686] T 00000000000000000000000000000000 P eb68676ca60d450fa5be3f864cf09dd3 [term 1 LEADER]: Becoming Leader. State: Replica: eb68676ca60d450fa5be3f864cf09dd3, State: Running, Role: LEADER
I20260806 06:30:42.401893 21212 sys_catalog.cc:564] T 00000000000000000000000000000000 P eb68676ca60d450fa5be3f864cf09dd3 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 06:30:42.401877 21219 consensus_queue.cc:237] T 00000000000000000000000000000000 P eb68676ca60d450fa5be3f864cf09dd3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb68676ca60d450fa5be3f864cf09dd3" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 45659 } }
I20260806 06:30:42.402204 21219 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb68676ca60d450fa5be3f864cf09dd3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader eb68676ca60d450fa5be3f864cf09dd3. Latest consensus state: current_term: 1 leader_uuid: "eb68676ca60d450fa5be3f864cf09dd3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb68676ca60d450fa5be3f864cf09dd3" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 45659 } } }
I20260806 06:30:42.402275 21219 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb68676ca60d450fa5be3f864cf09dd3 [sys.catalog]: This master's current role is: LEADER
I20260806 06:30:42.402454 21221 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb68676ca60d450fa5be3f864cf09dd3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eb68676ca60d450fa5be3f864cf09dd3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb68676ca60d450fa5be3f864cf09dd3" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 45659 } } }
I20260806 06:30:42.402518 21221 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb68676ca60d450fa5be3f864cf09dd3 [sys.catalog]: This master's current role is: LEADER
I20260806 06:30:42.403035 21228 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 06:30:42.403585 21228 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 06:30:42.405001 21228 catalog_manager.cc:1347] Generated new cluster ID: a35032461cbd49db9ab0e8f5ceb1cf44
I20260806 06:30:42.405051 21228 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 06:30:42.405464 21217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:42.399447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47607 (local address 127.21.12.190:45659)
0806 06:30:42.399616 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0806 06:30:42.399624 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 06:30:42.399647 (+ 23us) server_negotiation.cc:407] Connection header received
0806 06:30:42.400392 (+ 745us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:30:42.400397 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:30:42.400438 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:30:42.400536 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:30:42.400798 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:42.402146 (+ 1348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:42.403251 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:42.403818 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:42.403917 (+ 99us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:30:42.404692 (+ 775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:30:42.404695 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:30:42.404696 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 06:30:42.404885 (+ 189us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:30:42.404920 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 06:30:42.404936 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:30:42.405060 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0806 06:30:42.405237 (+ 177us) server_negotiation.cc:299] Negotiation successful
0806 06:30:42.405266 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":79,"threads_started":1}
I20260806 06:30:42.406324 21554 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 06:30:42.417096 21228 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 06:30:42.417711 21228 catalog_manager.cc:1504] Loading token signing keys...
I20260806 06:30:42.417814 21217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:42.410568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49097 (local address 127.21.12.190:45659)
0806 06:30:42.410602 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0806 06:30:42.410604 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:30:42.414429 (+ 3825us) server_negotiation.cc:407] Connection header received
0806 06:30:42.414476 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:30:42.414477 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:30:42.414488 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:30:42.414544 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:30:42.414659 (+ 115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:42.415388 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:42.416341 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:42.416676 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:42.416740 (+ 64us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:30:42.417513 (+ 773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:30:42.417514 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:30:42.417514 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 06:30:42.417519 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:30:42.417534 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 06:30:42.417538 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:30:42.417593 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0806 06:30:42.417682 (+ 89us) server_negotiation.cc:299] Negotiation successful
0806 06:30:42.417694 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260806 06:30:42.424319 21228 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eb68676ca60d450fa5be3f864cf09dd3: Generated new TSK 0
I20260806 06:30:42.424484 21228 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 06:30:42.429486 21554 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHhLhT/build/release/bin/kudu with pid 21115
W20260806 06:30:42.439816 21096 connection.cc:535] client connection to 127.21.12.190:45659 recv error: Network error: failed to read from TLS socket (remote: 127.21.12.190:45659): Connection reset by peer (error 104)
2026-08-06T06:30:42Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1401 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785997581049705-21554-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 06:30:42 dist-test-slave-4tdj krb5kdc[21254](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 06:30:42 dist-test-slave-4tdj krb5kdc[21254](info): set up 2 sockets
Aug 06 06:30:42 dist-test-slave-4tdj krb5kdc[21254](info): commencing operation
krb5kdc: starting...
W20260806 06:30:43.539883 21554 mini_kdc.cc:121] Time spent starting KDC: real 1.087s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785997581049705-21554-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785997581049705-21554-0/krb5kdc/test-user.keytab.
Aug 06 06:30:43 dist-test-slave-4tdj krb5kdc[21254](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997843, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T06:30:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T06:30:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.12.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.12.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.12.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.190.keytab.
Entry for principal kudu/127.21.12.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.190.keytab.
I20260806 06:30:43.653493 21554 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHhLhT/build/release/bin/kudu
/tmp/dist-test-taskFHhLhT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785997581049705-21554-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785997581049705-21554-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785997581049705-21554-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785997581049705-21554-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.12.190:38705
--webserver_interface=127.21.12.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.190.keytab
--principal=kudu/127.21.12.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.148:33515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.12.190:38705
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785997581049705-21554-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785997581049705-21554-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785997581049705-21554-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 06:30:43.733928 21271 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 06:30:43.734130 21271 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 06:30:43.734151 21271 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 06:30:43.735283 21271 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 06:30:43.735325 21271 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 06:30:43.735338 21271 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 06:30:43.735361 21271 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 06:30:43.735384 21271 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 06:30:43.736845 21271 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.148:33515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785997581049705-21554-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785997581049705-21554-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.12.190:38705
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.190.keytab
--principal=kudu/127.21.12.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.12.190:38705
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785997581049705-21554-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.12.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21271
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785997581049705-21554-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 05:11:02 UTC on ccycloud.quasar-hvzbbz.root.comops.site
build id 4569617
I20260806 06:30:43.737058 21271 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 06:30:43.737257 21271 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 06:30:43.741835 21280 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:30:43.741849 21277 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:30:43.742049 21271 server_base.cc:1034] running on GCE node
W20260806 06:30:43.742057 21278 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:30:43.742465 21271 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 06:30:43.742770 21271 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 06:30:43.743922 21271 hybrid_clock.cc:648] HybridClock initialized: now 1785997843743905 us; error 28 us; skew 500 ppm
Aug 06 06:30:43 dist-test-slave-4tdj krb5kdc[21254](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997843, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 06:30:43.749035 21271 init.cc:377] Logged in from keytab as kudu/127.21.12.190@KRBTEST.COM (short username kudu)
I20260806 06:30:43.749454 21271 webserver.cc:460] Webserver started at http://127.21.12.190:35969/ using document root <none> and password file <none>
I20260806 06:30:43.749652 21271 fs_manager.cc:359] Metadata directory not provided
I20260806 06:30:43.749699 21271 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 06:30:43.749825 21271 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 06:30:43.750903 21271 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785997581049705-21554-0/minicluster-data/master-0/data/instance:
uuid: "fe985294ca5142a9b8d6e2942ccb3cf2"
format_stamp: "Formatted at 2026-08-06 06:30:43 on dist-test-slave-4tdj"
server_key: "406530ef9cbb17e749b76456619a51bf"
server_key_iv: "a929d716c8871a355910c269a91add09"
server_key_version: "encryptionkey@0"
I20260806 06:30:43.751237 21271 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785997581049705-21554-0/minicluster-data/master-0/wal/instance:
uuid: "fe985294ca5142a9b8d6e2942ccb3cf2"
format_stamp: "Formatted at 2026-08-06 06:30:43 on dist-test-slave-4tdj"
server_key: "406530ef9cbb17e749b76456619a51bf"
server_key_iv: "a929d716c8871a355910c269a91add09"
server_key_version: "encryptionkey@0"
I20260806 06:30:43.752487 21271 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260806 06:30:43.753147 21292 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 06:30:43.753295 21271 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260806 06:30:43.753383 21271 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785997581049705-21554-0/minicluster-data/master-0/data,/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785997581049705-21554-0/minicluster-data/master-0/wal
uuid: "fe985294ca5142a9b8d6e2942ccb3cf2"
format_stamp: "Formatted at 2026-08-06 06:30:43 on dist-test-slave-4tdj"
server_key: "406530ef9cbb17e749b76456619a51bf"
server_key_iv: "a929d716c8871a355910c269a91add09"
server_key_version: "encryptionkey@0"
I20260806 06:30:43.753443 21271 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785997581049705-21554-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785997581049705-21554-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785997581049705-21554-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 06:30:43.756628 21271 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 06:30:43.758252 21271 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 06:30:43.758406 21271 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 06:30:43.762096 21271 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.190:38705
I20260806 06:30:43.762125 21371 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.190:38705 every 8 connection(s)
I20260806 06:30:43.762442 21271 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785997581049705-21554-0/minicluster-data/master-0/data/info.pb
I20260806 06:30:43.763345 21373 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 06:30:43.765429 21373 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe985294ca5142a9b8d6e2942ccb3cf2: Bootstrap starting.
I20260806 06:30:43.765939 21373 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fe985294ca5142a9b8d6e2942ccb3cf2: Neither blocks nor log segments found. Creating new log.
I20260806 06:30:43.766165 21373 log.cc:824] T 00000000000000000000000000000000 P fe985294ca5142a9b8d6e2942ccb3cf2: Log is configured to *not* fsync() on all Append() calls
I20260806 06:30:43.766794 21373 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe985294ca5142a9b8d6e2942ccb3cf2: No bootstrap required, opened a new log
I20260806 06:30:43.767990 21554 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHhLhT/build/release/bin/kudu as pid 21271
I20260806 06:30:43.768087 21554 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785997581049705-21554-0/minicluster-data/master-0/wal/instance
I20260806 06:30:43.768257 21554 external_mini_cluster.cc:1428] Setting key 6a4f1ac5b6913dcd639d4e7c4bb07b95
I20260806 06:30:43.768358 21373 raft_consensus.cc:348] T 00000000000000000000000000000000 P fe985294ca5142a9b8d6e2942ccb3cf2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe985294ca5142a9b8d6e2942ccb3cf2" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 38705 } }
I20260806 06:30:43.768492 21373 raft_consensus.cc:374] T 00000000000000000000000000000000 P fe985294ca5142a9b8d6e2942ccb3cf2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 06:30:43.768525 21373 raft_consensus.cc:729] T 00000000000000000000000000000000 P fe985294ca5142a9b8d6e2942ccb3cf2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe985294ca5142a9b8d6e2942ccb3cf2, State: Initialized, Role: FOLLOWER
I20260806 06:30:43.768662 21373 consensus_queue.cc:260] T 00000000000000000000000000000000 P fe985294ca5142a9b8d6e2942ccb3cf2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe985294ca5142a9b8d6e2942ccb3cf2" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 38705 } }
I20260806 06:30:43.768740 21373 raft_consensus.cc:388] T 00000000000000000000000000000000 P fe985294ca5142a9b8d6e2942ccb3cf2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 06:30:43.768770 21373 raft_consensus.cc:482] T 00000000000000000000000000000000 P fe985294ca5142a9b8d6e2942ccb3cf2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 06:30:43.768812 21373 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fe985294ca5142a9b8d6e2942ccb3cf2 [term 0 FOLLOWER]: Advancing to term 1
I20260806 06:30:43.769308 21373 raft_consensus.cc:504] T 00000000000000000000000000000000 P fe985294ca5142a9b8d6e2942ccb3cf2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe985294ca5142a9b8d6e2942ccb3cf2" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 38705 } }
I20260806 06:30:43.769414 21373 leader_election.cc:302] T 00000000000000000000000000000000 P fe985294ca5142a9b8d6e2942ccb3cf2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fe985294ca5142a9b8d6e2942ccb3cf2; no voters:
I20260806 06:30:43.769563 21373 leader_election.cc:288] T 00000000000000000000000000000000 P fe985294ca5142a9b8d6e2942ccb3cf2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 06:30:43.769646 21383 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fe985294ca5142a9b8d6e2942ccb3cf2 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 06:30:43.769774 21383 raft_consensus.cc:686] T 00000000000000000000000000000000 P fe985294ca5142a9b8d6e2942ccb3cf2 [term 1 LEADER]: Becoming Leader. State: Replica: fe985294ca5142a9b8d6e2942ccb3cf2, State: Running, Role: LEADER
I20260806 06:30:43.769862 21383 consensus_queue.cc:237] T 00000000000000000000000000000000 P fe985294ca5142a9b8d6e2942ccb3cf2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe985294ca5142a9b8d6e2942ccb3cf2" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 38705 } }
I20260806 06:30:43.770084 21373 sys_catalog.cc:564] T 00000000000000000000000000000000 P fe985294ca5142a9b8d6e2942ccb3cf2 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 06:30:43.770395 21385 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe985294ca5142a9b8d6e2942ccb3cf2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fe985294ca5142a9b8d6e2942ccb3cf2. Latest consensus state: current_term: 1 leader_uuid: "fe985294ca5142a9b8d6e2942ccb3cf2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe985294ca5142a9b8d6e2942ccb3cf2" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 38705 } } }
I20260806 06:30:43.770489 21385 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe985294ca5142a9b8d6e2942ccb3cf2 [sys.catalog]: This master's current role is: LEADER
I20260806 06:30:43.770376 21384 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe985294ca5142a9b8d6e2942ccb3cf2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fe985294ca5142a9b8d6e2942ccb3cf2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe985294ca5142a9b8d6e2942ccb3cf2" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 38705 } } }
I20260806 06:30:43.770646 21384 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe985294ca5142a9b8d6e2942ccb3cf2 [sys.catalog]: This master's current role is: LEADER
I20260806 06:30:43.770726 21382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:43.768735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59431 (local address 127.21.12.190:38705)
0806 06:30:43.768927 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0806 06:30:43.768936 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 06:30:43.768967 (+ 31us) server_negotiation.cc:407] Connection header received
0806 06:30:43.769275 (+ 308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:30:43.769281 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:30:43.769305 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:30:43.769420 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:30:43.770123 (+ 703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:30:43.770127 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:30:43.770127 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 06:30:43.770133 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:30:43.770182 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 06:30:43.770198 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:30:43.770264 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0806 06:30:43.770460 (+ 196us) server_negotiation.cc:299] Negotiation successful
0806 06:30:43.770502 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":91,"threads_started":1}
I20260806 06:30:43.771271 21389 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 06:30:43.771657 21554 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 06:30:43.771806 21389 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 06:30:43.773404 21389 catalog_manager.cc:1347] Generated new cluster ID: b6e63b828c8646efb17aab0e1db1f508
I20260806 06:30:43.773459 21389 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 06:30:43.774447 21382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:43.773282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37427 (local address 127.21.12.190:38705)
0806 06:30:43.773376 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0806 06:30:43.773378 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:30:43.773494 (+ 116us) server_negotiation.cc:407] Connection header received
0806 06:30:43.773610 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:30:43.773611 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:30:43.773615 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:30:43.773691 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:30:43.774222 (+ 531us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:30:43.774223 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:30:43.774224 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 06:30:43.774228 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:30:43.774245 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 06:30:43.774249 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:30:43.774336 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0806 06:30:43.774376 (+ 40us) server_negotiation.cc:299] Negotiation successful
0806 06:30:43.774389 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260806 06:30:43.782627 21389 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 06:30:43.782781 21389 catalog_manager.cc:1504] Loading token signing keys...
I20260806 06:30:43.788019 21389 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fe985294ca5142a9b8d6e2942ccb3cf2: Generated new TSK 0
I20260806 06:30:43.788185 21389 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 06:30:43.794736 21554 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHhLhT/build/release/bin/kudu with pid 21271
2026-08-06T06:30:43Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1367 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785997581049705-21554-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 06:30:43 dist-test-slave-4tdj krb5kdc[21415](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 06:30:43 dist-test-slave-4tdj krb5kdc[21415](info): set up 2 sockets
Aug 06 06:30:43 dist-test-slave-4tdj krb5kdc[21415](info): commencing operation
krb5kdc: starting...
W20260806 06:30:44.913300 21554 mini_kdc.cc:121] Time spent starting KDC: real 1.093s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785997581049705-21554-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785997581049705-21554-0/krb5kdc/test-user.keytab.
Aug 06 06:30:44 dist-test-slave-4tdj krb5kdc[21415](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997844, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T06:30:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T06:30:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.12.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.12.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.12.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.190.keytab.
Entry for principal kudu/127.21.12.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.190.keytab.
I20260806 06:30:45.024958 21554 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHhLhT/build/release/bin/kudu
/tmp/dist-test-taskFHhLhT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785997581049705-21554-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785997581049705-21554-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785997581049705-21554-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785997581049705-21554-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.12.190:40989
--webserver_interface=127.21.12.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.190.keytab
--principal=kudu/127.21.12.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.148:36557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.12.190:40989
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785997581049705-21554-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785997581049705-21554-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785997581049705-21554-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 06:30:45.105975 21431 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 06:30:45.106177 21431 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 06:30:45.106201 21431 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 06:30:45.107352 21431 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 06:30:45.107398 21431 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 06:30:45.107414 21431 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 06:30:45.107441 21431 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 06:30:45.107476 21431 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 06:30:45.108920 21431 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.148:36557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785997581049705-21554-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785997581049705-21554-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.12.190:40989
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.190.keytab
--principal=kudu/127.21.12.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.12.190:40989
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785997581049705-21554-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.12.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21431
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785997581049705-21554-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 05:11:02 UTC on ccycloud.quasar-hvzbbz.root.comops.site
build id 4569617
I20260806 06:30:45.109143 21431 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 06:30:45.109349 21431 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 06:30:45.114041 21440 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:30:45.114054 21437 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:30:45.114166 21438 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:30:45.114224 21431 server_base.cc:1034] running on GCE node
I20260806 06:30:45.114526 21431 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 06:30:45.114822 21431 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 06:30:45.115955 21431 hybrid_clock.cc:648] HybridClock initialized: now 1785997845115946 us; error 31 us; skew 500 ppm
Aug 06 06:30:45 dist-test-slave-4tdj krb5kdc[21415](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997845, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 06:30:45.118397 21431 init.cc:377] Logged in from keytab as kudu/127.21.12.190@KRBTEST.COM (short username kudu)
I20260806 06:30:45.118796 21431 webserver.cc:460] Webserver started at http://127.21.12.190:43633/ using document root <none> and password file <none>
I20260806 06:30:45.119004 21431 fs_manager.cc:359] Metadata directory not provided
I20260806 06:30:45.119045 21431 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 06:30:45.119166 21431 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 06:30:45.120147 21431 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785997581049705-21554-0/minicluster-data/master-0/data/instance:
uuid: "fffc4def978b464e94b4c9de88ff83db"
format_stamp: "Formatted at 2026-08-06 06:30:45 on dist-test-slave-4tdj"
server_key: "b9869d0add12c5e7be163a91fa8dfbb1"
server_key_iv: "171ab77ca1ae641676acdc1a85d00eaa"
server_key_version: "encryptionkey@0"
I20260806 06:30:45.120463 21431 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785997581049705-21554-0/minicluster-data/master-0/wal/instance:
uuid: "fffc4def978b464e94b4c9de88ff83db"
format_stamp: "Formatted at 2026-08-06 06:30:45 on dist-test-slave-4tdj"
server_key: "b9869d0add12c5e7be163a91fa8dfbb1"
server_key_iv: "171ab77ca1ae641676acdc1a85d00eaa"
server_key_version: "encryptionkey@0"
I20260806 06:30:45.121694 21431 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260806 06:30:45.122346 21453 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 06:30:45.122488 21431 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260806 06:30:45.122571 21431 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785997581049705-21554-0/minicluster-data/master-0/data,/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785997581049705-21554-0/minicluster-data/master-0/wal
uuid: "fffc4def978b464e94b4c9de88ff83db"
format_stamp: "Formatted at 2026-08-06 06:30:45 on dist-test-slave-4tdj"
server_key: "b9869d0add12c5e7be163a91fa8dfbb1"
server_key_iv: "171ab77ca1ae641676acdc1a85d00eaa"
server_key_version: "encryptionkey@0"
I20260806 06:30:45.122637 21431 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785997581049705-21554-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785997581049705-21554-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785997581049705-21554-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 06:30:45.140252 21431 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 06:30:45.142228 21431 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 06:30:45.142403 21431 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 06:30:45.145984 21431 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.190:40989
I20260806 06:30:45.146034 21532 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.190:40989 every 8 connection(s)
I20260806 06:30:45.146358 21431 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785997581049705-21554-0/minicluster-data/master-0/data/info.pb
I20260806 06:30:45.147192 21533 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 06:30:45.149401 21554 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHhLhT/build/release/bin/kudu as pid 21431
I20260806 06:30:45.149428 21533 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fffc4def978b464e94b4c9de88ff83db: Bootstrap starting.
I20260806 06:30:45.149507 21554 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785997581049705-21554-0/minicluster-data/master-0/wal/instance
I20260806 06:30:45.149681 21554 external_mini_cluster.cc:1428] Setting key 93acb720f738efcd943c10bbd0a7d19b
I20260806 06:30:45.150020 21533 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fffc4def978b464e94b4c9de88ff83db: Neither blocks nor log segments found. Creating new log.
I20260806 06:30:45.150369 21533 log.cc:824] T 00000000000000000000000000000000 P fffc4def978b464e94b4c9de88ff83db: Log is configured to *not* fsync() on all Append() calls
I20260806 06:30:45.151062 21533 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fffc4def978b464e94b4c9de88ff83db: No bootstrap required, opened a new log
I20260806 06:30:45.153177 21533 raft_consensus.cc:348] T 00000000000000000000000000000000 P fffc4def978b464e94b4c9de88ff83db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fffc4def978b464e94b4c9de88ff83db" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 40989 } }
I20260806 06:30:45.153343 21533 raft_consensus.cc:374] T 00000000000000000000000000000000 P fffc4def978b464e94b4c9de88ff83db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 06:30:45.153374 21533 raft_consensus.cc:729] T 00000000000000000000000000000000 P fffc4def978b464e94b4c9de88ff83db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fffc4def978b464e94b4c9de88ff83db, State: Initialized, Role: FOLLOWER
I20260806 06:30:45.153496 21533 consensus_queue.cc:260] T 00000000000000000000000000000000 P fffc4def978b464e94b4c9de88ff83db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fffc4def978b464e94b4c9de88ff83db" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 40989 } }
I20260806 06:30:45.153566 21533 raft_consensus.cc:388] T 00000000000000000000000000000000 P fffc4def978b464e94b4c9de88ff83db [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 06:30:45.153600 21533 raft_consensus.cc:482] T 00000000000000000000000000000000 P fffc4def978b464e94b4c9de88ff83db [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 06:30:45.153643 21533 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fffc4def978b464e94b4c9de88ff83db [term 0 FOLLOWER]: Advancing to term 1
I20260806 06:30:45.154173 21533 raft_consensus.cc:504] T 00000000000000000000000000000000 P fffc4def978b464e94b4c9de88ff83db [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fffc4def978b464e94b4c9de88ff83db" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 40989 } }
I20260806 06:30:45.154280 21533 leader_election.cc:302] T 00000000000000000000000000000000 P fffc4def978b464e94b4c9de88ff83db [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fffc4def978b464e94b4c9de88ff83db; no voters:
I20260806 06:30:45.154476 21533 leader_election.cc:288] T 00000000000000000000000000000000 P fffc4def978b464e94b4c9de88ff83db [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 06:30:45.154512 21539 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fffc4def978b464e94b4c9de88ff83db [term 1 FOLLOWER]: Leader election won for term 1
I20260806 06:30:45.154649 21539 raft_consensus.cc:686] T 00000000000000000000000000000000 P fffc4def978b464e94b4c9de88ff83db [term 1 LEADER]: Becoming Leader. State: Replica: fffc4def978b464e94b4c9de88ff83db, State: Running, Role: LEADER
I20260806 06:30:45.154736 21533 sys_catalog.cc:564] T 00000000000000000000000000000000 P fffc4def978b464e94b4c9de88ff83db [sys.catalog]: configured and running, proceeding with master startup.
I20260806 06:30:45.154745 21539 consensus_queue.cc:237] T 00000000000000000000000000000000 P fffc4def978b464e94b4c9de88ff83db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fffc4def978b464e94b4c9de88ff83db" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 40989 } }
I20260806 06:30:45.155097 21539 sys_catalog.cc:455] T 00000000000000000000000000000000 P fffc4def978b464e94b4c9de88ff83db [sys.catalog]: SysCatalogTable state changed. Reason: New leader fffc4def978b464e94b4c9de88ff83db. Latest consensus state: current_term: 1 leader_uuid: "fffc4def978b464e94b4c9de88ff83db" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fffc4def978b464e94b4c9de88ff83db" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 40989 } } }
I20260806 06:30:45.155176 21539 sys_catalog.cc:458] T 00000000000000000000000000000000 P fffc4def978b464e94b4c9de88ff83db [sys.catalog]: This master's current role is: LEADER
I20260806 06:30:45.155336 21542 sys_catalog.cc:455] T 00000000000000000000000000000000 P fffc4def978b464e94b4c9de88ff83db [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fffc4def978b464e94b4c9de88ff83db" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fffc4def978b464e94b4c9de88ff83db" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 40989 } } }
I20260806 06:30:45.155390 21542 sys_catalog.cc:458] T 00000000000000000000000000000000 P fffc4def978b464e94b4c9de88ff83db [sys.catalog]: This master's current role is: LEADER
I20260806 06:30:45.155658 21547 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 06:30:45.156118 21547 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Aug 06 06:30:45 dist-test-slave-4tdj krb5kdc[21415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997844, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.12.190@KRBTEST.COM
I20260806 06:30:45.157744 21547 catalog_manager.cc:1347] Generated new cluster ID: 40e65cb413104975af74d8f84ffa0ee4
I20260806 06:30:45.157799 21547 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 06:30:45.160838 21537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:45.150165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59439 (local address 127.21.12.190:40989)
0806 06:30:45.150364 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0806 06:30:45.150374 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 06:30:45.150398 (+ 24us) server_negotiation.cc:407] Connection header received
0806 06:30:45.150853 (+ 455us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:30:45.150860 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:30:45.150907 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:30:45.151018 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:30:45.151768 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:45.152663 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:45.153693 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:45.154103 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:45.154184 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:30:45.156917 (+ 2733us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:30:45.156921 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:30:45.156922 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 06:30:45.156933 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:30:45.158825 (+ 1892us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:30:45.159228 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:30:45.159231 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:30:45.159232 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:30:45.159335 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:30:45.159533 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:30:45.159534 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:30:45.159534 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:30:45.159828 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:30:45.159974 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0806 06:30:45.160489 (+ 515us) server_negotiation.cc:299] Negotiation successful
0806 06:30:45.160624 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":74,"threads_started":1}
I20260806 06:30:45.161813 21554 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 06:30:45.169566 21537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:45.163148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57539 (local address 127.21.12.190:40989)
0806 06:30:45.163189 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0806 06:30:45.163191 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:30:45.163221 (+ 30us) server_negotiation.cc:407] Connection header received
0806 06:30:45.163274 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:30:45.163275 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:30:45.163284 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:30:45.163347 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:30:45.163894 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:45.164618 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:45.165543 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:45.165854 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:45.165924 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:30:45.167119 (+ 1195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:30:45.167120 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:30:45.167121 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 06:30:45.167128 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:30:45.168563 (+ 1435us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:30:45.168888 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:30:45.168889 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:30:45.168889 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:30:45.168943 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:30:45.169107 (+ 164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:30:45.169108 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:30:45.169108 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:30:45.169235 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:30:45.169323 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0806 06:30:45.169409 (+ 86us) server_negotiation.cc:299] Negotiation successful
0806 06:30:45.169479 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260806 06:30:45.227996 21547 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 06:30:45.228757 21547 catalog_manager.cc:1504] Loading token signing keys...
I20260806 06:30:45.236208 21547 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fffc4def978b464e94b4c9de88ff83db: Generated new TSK 0
I20260806 06:30:45.236426 21547 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 06:30:45.238884 21554 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHhLhT/build/release/bin/kudu with pid 21431
2026-08-06T06:30:45Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1440 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785997581049705-21554-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 06:30:45 dist-test-slave-4tdj krb5kdc[21574](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 06:30:45 dist-test-slave-4tdj krb5kdc[21574](info): set up 2 sockets
Aug 06 06:30:45 dist-test-slave-4tdj krb5kdc[21574](info): commencing operation
krb5kdc: starting...
W20260806 06:30:46.365298 21554 mini_kdc.cc:121] Time spent starting KDC: real 1.105s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785997581049705-21554-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785997581049705-21554-0/krb5kdc/test-user.keytab.
Aug 06 06:30:46 dist-test-slave-4tdj krb5kdc[21574](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997846, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T06:30:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T06:30:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.12.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.12.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.12.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.190.keytab.
Entry for principal kudu/127.21.12.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.190.keytab.
I20260806 06:30:46.481033 21554 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHhLhT/build/release/bin/kudu
/tmp/dist-test-taskFHhLhT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785997581049705-21554-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785997581049705-21554-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785997581049705-21554-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785997581049705-21554-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.12.190:33549
--webserver_interface=127.21.12.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.190.keytab
--principal=kudu/127.21.12.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.148:43001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.12.190:33549
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785997581049705-21554-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785997581049705-21554-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785997581049705-21554-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 06:30:46.561079 21594 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 06:30:46.561273 21594 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 06:30:46.561295 21594 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 06:30:46.562427 21594 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 06:30:46.562465 21594 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 06:30:46.562496 21594 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 06:30:46.562513 21594 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 06:30:46.562536 21594 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 06:30:46.564006 21594 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.148:43001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785997581049705-21554-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785997581049705-21554-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.12.190:33549
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.190.keytab
--principal=kudu/127.21.12.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.12.190:33549
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785997581049705-21554-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.12.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21594
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785997581049705-21554-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 05:11:02 UTC on ccycloud.quasar-hvzbbz.root.comops.site
build id 4569617
I20260806 06:30:46.564235 21594 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 06:30:46.564430 21594 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 06:30:46.569208 21606 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:30:46.569247 21605 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:30:46.569208 21608 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:30:46.569334 21594 server_base.cc:1034] running on GCE node
I20260806 06:30:46.569664 21594 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 06:30:46.569960 21594 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 06:30:46.571110 21594 hybrid_clock.cc:648] HybridClock initialized: now 1785997846571093 us; error 28 us; skew 500 ppm
Aug 06 06:30:46 dist-test-slave-4tdj krb5kdc[21574](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997846, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 06:30:46.573472 21594 init.cc:377] Logged in from keytab as kudu/127.21.12.190@KRBTEST.COM (short username kudu)
I20260806 06:30:46.573915 21594 webserver.cc:460] Webserver started at http://127.21.12.190:34869/ using document root <none> and password file <none>
I20260806 06:30:46.574128 21594 fs_manager.cc:359] Metadata directory not provided
I20260806 06:30:46.574177 21594 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 06:30:46.574323 21594 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 06:30:46.575344 21594 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785997581049705-21554-0/minicluster-data/master-0/data/instance:
uuid: "ab908c03e4ed499f995e4a03e75c0db9"
format_stamp: "Formatted at 2026-08-06 06:30:46 on dist-test-slave-4tdj"
server_key: "02f0c7dbfe00d4c18535cd5638da970a"
server_key_iv: "83432d0bfac36dd05a6569312ee15ba6"
server_key_version: "encryptionkey@0"
I20260806 06:30:46.575671 21594 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785997581049705-21554-0/minicluster-data/master-0/wal/instance:
uuid: "ab908c03e4ed499f995e4a03e75c0db9"
format_stamp: "Formatted at 2026-08-06 06:30:46 on dist-test-slave-4tdj"
server_key: "02f0c7dbfe00d4c18535cd5638da970a"
server_key_iv: "83432d0bfac36dd05a6569312ee15ba6"
server_key_version: "encryptionkey@0"
I20260806 06:30:46.576864 21594 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260806 06:30:46.577473 21617 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 06:30:46.577642 21594 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260806 06:30:46.577729 21594 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785997581049705-21554-0/minicluster-data/master-0/data,/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785997581049705-21554-0/minicluster-data/master-0/wal
uuid: "ab908c03e4ed499f995e4a03e75c0db9"
format_stamp: "Formatted at 2026-08-06 06:30:46 on dist-test-slave-4tdj"
server_key: "02f0c7dbfe00d4c18535cd5638da970a"
server_key_iv: "83432d0bfac36dd05a6569312ee15ba6"
server_key_version: "encryptionkey@0"
I20260806 06:30:46.577790 21594 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785997581049705-21554-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785997581049705-21554-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785997581049705-21554-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 06:30:46.596552 21594 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 06:30:46.598169 21594 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 06:30:46.598326 21594 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 06:30:46.601907 21594 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.190:33549
I20260806 06:30:46.601964 21683 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.190:33549 every 8 connection(s)
I20260806 06:30:46.602267 21594 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785997581049705-21554-0/minicluster-data/master-0/data/info.pb
I20260806 06:30:46.603091 21684 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 06:30:46.605253 21684 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ab908c03e4ed499f995e4a03e75c0db9: Bootstrap starting.
I20260806 06:30:46.605747 21554 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHhLhT/build/release/bin/kudu as pid 21594
I20260806 06:30:46.605780 21684 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ab908c03e4ed499f995e4a03e75c0db9: Neither blocks nor log segments found. Creating new log.
I20260806 06:30:46.605852 21554 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785997581049705-21554-0/minicluster-data/master-0/wal/instance
I20260806 06:30:46.606024 21554 external_mini_cluster.cc:1428] Setting key 28daedf1d42afeebaf1fe77c12f0bd20
I20260806 06:30:46.606034 21684 log.cc:824] T 00000000000000000000000000000000 P ab908c03e4ed499f995e4a03e75c0db9: Log is configured to *not* fsync() on all Append() calls
I20260806 06:30:46.606787 21684 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ab908c03e4ed499f995e4a03e75c0db9: No bootstrap required, opened a new log
I20260806 06:30:46.608726 21684 raft_consensus.cc:348] T 00000000000000000000000000000000 P ab908c03e4ed499f995e4a03e75c0db9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab908c03e4ed499f995e4a03e75c0db9" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 33549 } }
I20260806 06:30:46.608855 21684 raft_consensus.cc:374] T 00000000000000000000000000000000 P ab908c03e4ed499f995e4a03e75c0db9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 06:30:46.608888 21684 raft_consensus.cc:729] T 00000000000000000000000000000000 P ab908c03e4ed499f995e4a03e75c0db9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab908c03e4ed499f995e4a03e75c0db9, State: Initialized, Role: FOLLOWER
I20260806 06:30:46.609002 21684 consensus_queue.cc:260] T 00000000000000000000000000000000 P ab908c03e4ed499f995e4a03e75c0db9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab908c03e4ed499f995e4a03e75c0db9" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 33549 } }
I20260806 06:30:46.609061 21684 raft_consensus.cc:388] T 00000000000000000000000000000000 P ab908c03e4ed499f995e4a03e75c0db9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 06:30:46.609092 21684 raft_consensus.cc:482] T 00000000000000000000000000000000 P ab908c03e4ed499f995e4a03e75c0db9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 06:30:46.609136 21684 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ab908c03e4ed499f995e4a03e75c0db9 [term 0 FOLLOWER]: Advancing to term 1
I20260806 06:30:46.609632 21684 raft_consensus.cc:504] T 00000000000000000000000000000000 P ab908c03e4ed499f995e4a03e75c0db9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab908c03e4ed499f995e4a03e75c0db9" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 33549 } }
I20260806 06:30:46.609735 21684 leader_election.cc:302] T 00000000000000000000000000000000 P ab908c03e4ed499f995e4a03e75c0db9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ab908c03e4ed499f995e4a03e75c0db9; no voters:
I20260806 06:30:46.609884 21684 leader_election.cc:288] T 00000000000000000000000000000000 P ab908c03e4ed499f995e4a03e75c0db9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 06:30:46.609907 21692 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ab908c03e4ed499f995e4a03e75c0db9 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 06:30:46.610133 21684 sys_catalog.cc:564] T 00000000000000000000000000000000 P ab908c03e4ed499f995e4a03e75c0db9 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 06:30:46.610368 21692 raft_consensus.cc:686] T 00000000000000000000000000000000 P ab908c03e4ed499f995e4a03e75c0db9 [term 1 LEADER]: Becoming Leader. State: Replica: ab908c03e4ed499f995e4a03e75c0db9, State: Running, Role: LEADER
I20260806 06:30:46.610519 21692 consensus_queue.cc:237] T 00000000000000000000000000000000 P ab908c03e4ed499f995e4a03e75c0db9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab908c03e4ed499f995e4a03e75c0db9" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 33549 } }
I20260806 06:30:46.610960 21695 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab908c03e4ed499f995e4a03e75c0db9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ab908c03e4ed499f995e4a03e75c0db9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab908c03e4ed499f995e4a03e75c0db9" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 33549 } } }
I20260806 06:30:46.611068 21695 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab908c03e4ed499f995e4a03e75c0db9 [sys.catalog]: This master's current role is: LEADER
I20260806 06:30:46.611119 21690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:46.606482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50783 (local address 127.21.12.190:33549)
0806 06:30:46.606646 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0806 06:30:46.606656 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 06:30:46.606690 (+ 34us) server_negotiation.cc:407] Connection header received
0806 06:30:46.607162 (+ 472us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:30:46.607169 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:30:46.607218 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:30:46.607338 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:30:46.607560 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:46.608424 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:46.609307 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:46.609699 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:46.609794 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:30:46.610462 (+ 668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:30:46.610466 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:30:46.610466 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 06:30:46.610656 (+ 190us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:30:46.610697 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 06:30:46.610713 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:30:46.610774 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0806 06:30:46.610833 (+ 59us) server_negotiation.cc:299] Negotiation successful
0806 06:30:46.610895 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":73,"threads_started":1}
I20260806 06:30:46.611285 21694 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab908c03e4ed499f995e4a03e75c0db9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ab908c03e4ed499f995e4a03e75c0db9. Latest consensus state: current_term: 1 leader_uuid: "ab908c03e4ed499f995e4a03e75c0db9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab908c03e4ed499f995e4a03e75c0db9" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 33549 } } }
I20260806 06:30:46.611356 21694 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab908c03e4ed499f995e4a03e75c0db9 [sys.catalog]: This master's current role is: LEADER
W20260806 06:30:46.611948 21713 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ab908c03e4ed499f995e4a03e75c0db9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 06:30:46.612003 21713 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 06:30:46.611986 21554 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 06:30:46.612568 21707 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 06:30:46.612824 21707 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 06:30:46.614365 21707 catalog_manager.cc:1347] Generated new cluster ID: e53da67030cf429d9e2555f54e67a100
I20260806 06:30:46.614415 21707 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 06:30:46.616567 21690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:46.613303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39727 (local address 127.21.12.190:33549)
0806 06:30:46.613337 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0806 06:30:46.613339 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:30:46.613473 (+ 134us) server_negotiation.cc:407] Connection header received
0806 06:30:46.613558 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:30:46.613558 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:30:46.613568 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:30:46.613593 (+ 25us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:30:46.613768 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:46.614453 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:46.615435 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:46.615719 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:46.615774 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:30:46.616367 (+ 593us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:30:46.616369 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:30:46.616369 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 06:30:46.616373 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:30:46.616385 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 06:30:46.616389 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:30:46.616427 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0806 06:30:46.616484 (+ 57us) server_negotiation.cc:299] Negotiation successful
0806 06:30:46.616502 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260806 06:30:46.632354 21707 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 06:30:46.632825 21707 catalog_manager.cc:1504] Loading token signing keys...
I20260806 06:30:46.637251 21707 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ab908c03e4ed499f995e4a03e75c0db9: Generated new TSK 0
I20260806 06:30:46.637399 21707 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 06:30:46.640466 21554 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHhLhT/build/release/bin/kudu with pid 21594
2026-08-06T06:30:46Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1409 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785997581049705-21554-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 06:30:46 dist-test-slave-4tdj krb5kdc[21725](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 06:30:46 dist-test-slave-4tdj krb5kdc[21725](info): set up 2 sockets
Aug 06 06:30:46 dist-test-slave-4tdj krb5kdc[21725](info): commencing operation
krb5kdc: starting...
W20260806 06:30:47.774019 21554 mini_kdc.cc:121] Time spent starting KDC: real 1.104s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785997581049705-21554-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785997581049705-21554-0/krb5kdc/test-user.keytab.
Aug 06 06:30:47 dist-test-slave-4tdj krb5kdc[21725](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997847, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T06:30:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T06:30:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.12.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.12.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.12.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.190.keytab.
Entry for principal kudu/127.21.12.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.190.keytab.
I20260806 06:30:47.887233 21554 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHhLhT/build/release/bin/kudu
/tmp/dist-test-taskFHhLhT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785997581049705-21554-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785997581049705-21554-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785997581049705-21554-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785997581049705-21554-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.12.190:39891
--webserver_interface=127.21.12.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.190.keytab
--principal=kudu/127.21.12.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.148:34251
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.12.190:39891
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785997581049705-21554-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785997581049705-21554-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785997581049705-21554-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 06:30:47.965960 21744 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 06:30:47.966164 21744 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 06:30:47.966195 21744 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 06:30:47.967324 21744 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 06:30:47.967365 21744 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 06:30:47.967382 21744 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 06:30:47.967398 21744 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 06:30:47.967413 21744 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 06:30:47.968838 21744 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.148:34251
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785997581049705-21554-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785997581049705-21554-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.12.190:39891
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.190.keytab
--principal=kudu/127.21.12.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.12.190:39891
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785997581049705-21554-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.12.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21744
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785997581049705-21554-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 05:11:02 UTC on ccycloud.quasar-hvzbbz.root.comops.site
build id 4569617
I20260806 06:30:47.969069 21744 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 06:30:47.969285 21744 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 06:30:47.973881 21755 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:30:47.973938 21744 server_base.cc:1034] running on GCE node
W20260806 06:30:47.973909 21753 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:30:47.973882 21752 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:30:47.974428 21744 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 06:30:47.974699 21744 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 06:30:47.975837 21744 hybrid_clock.cc:648] HybridClock initialized: now 1785997847975825 us; error 29 us; skew 500 ppm
Aug 06 06:30:47 dist-test-slave-4tdj krb5kdc[21725](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997847, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 06:30:47.978093 21744 init.cc:377] Logged in from keytab as kudu/127.21.12.190@KRBTEST.COM (short username kudu)
I20260806 06:30:47.978523 21744 webserver.cc:460] Webserver started at http://127.21.12.190:40181/ using document root <none> and password file <none>
I20260806 06:30:47.978734 21744 fs_manager.cc:359] Metadata directory not provided
I20260806 06:30:47.978781 21744 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 06:30:47.978904 21744 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 06:30:47.979908 21744 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785997581049705-21554-0/minicluster-data/master-0/data/instance:
uuid: "8a8f9156030c47c48add5e8125a7ec65"
format_stamp: "Formatted at 2026-08-06 06:30:47 on dist-test-slave-4tdj"
server_key: "d3189aeb2cc334509f992d4a6ffcf0cb"
server_key_iv: "e5fda6494eedf954c053faa21de57c6f"
server_key_version: "encryptionkey@0"
I20260806 06:30:47.980233 21744 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785997581049705-21554-0/minicluster-data/master-0/wal/instance:
uuid: "8a8f9156030c47c48add5e8125a7ec65"
format_stamp: "Formatted at 2026-08-06 06:30:47 on dist-test-slave-4tdj"
server_key: "d3189aeb2cc334509f992d4a6ffcf0cb"
server_key_iv: "e5fda6494eedf954c053faa21de57c6f"
server_key_version: "encryptionkey@0"
I20260806 06:30:47.981392 21744 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260806 06:30:47.982018 21764 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 06:30:47.982158 21744 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260806 06:30:47.982231 21744 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785997581049705-21554-0/minicluster-data/master-0/data,/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785997581049705-21554-0/minicluster-data/master-0/wal
uuid: "8a8f9156030c47c48add5e8125a7ec65"
format_stamp: "Formatted at 2026-08-06 06:30:47 on dist-test-slave-4tdj"
server_key: "d3189aeb2cc334509f992d4a6ffcf0cb"
server_key_iv: "e5fda6494eedf954c053faa21de57c6f"
server_key_version: "encryptionkey@0"
I20260806 06:30:47.982326 21744 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785997581049705-21554-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785997581049705-21554-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785997581049705-21554-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 06:30:47.985438 21744 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 06:30:47.987120 21744 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 06:30:47.987241 21744 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 06:30:47.990868 21744 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.190:39891
I20260806 06:30:47.990922 21838 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.190:39891 every 8 connection(s)
I20260806 06:30:47.991246 21744 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785997581049705-21554-0/minicluster-data/master-0/data/info.pb
I20260806 06:30:47.992153 21840 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 06:30:47.994325 21840 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a8f9156030c47c48add5e8125a7ec65: Bootstrap starting.
I20260806 06:30:47.994886 21840 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8a8f9156030c47c48add5e8125a7ec65: Neither blocks nor log segments found. Creating new log.
I20260806 06:30:47.995144 21840 log.cc:824] T 00000000000000000000000000000000 P 8a8f9156030c47c48add5e8125a7ec65: Log is configured to *not* fsync() on all Append() calls
I20260806 06:30:47.995786 21840 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a8f9156030c47c48add5e8125a7ec65: No bootstrap required, opened a new log
I20260806 06:30:47.997303 21840 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8a8f9156030c47c48add5e8125a7ec65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a8f9156030c47c48add5e8125a7ec65" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 39891 } }
I20260806 06:30:47.997437 21840 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8a8f9156030c47c48add5e8125a7ec65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 06:30:47.997469 21840 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8a8f9156030c47c48add5e8125a7ec65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a8f9156030c47c48add5e8125a7ec65, State: Initialized, Role: FOLLOWER
I20260806 06:30:47.997603 21840 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8a8f9156030c47c48add5e8125a7ec65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a8f9156030c47c48add5e8125a7ec65" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 39891 } }
I20260806 06:30:47.997669 21840 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8a8f9156030c47c48add5e8125a7ec65 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 06:30:47.997704 21840 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8a8f9156030c47c48add5e8125a7ec65 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 06:30:47.997749 21840 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8a8f9156030c47c48add5e8125a7ec65 [term 0 FOLLOWER]: Advancing to term 1
I20260806 06:30:47.998593 21840 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8a8f9156030c47c48add5e8125a7ec65 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a8f9156030c47c48add5e8125a7ec65" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 39891 } }
I20260806 06:30:47.998696 21840 leader_election.cc:302] T 00000000000000000000000000000000 P 8a8f9156030c47c48add5e8125a7ec65 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8a8f9156030c47c48add5e8125a7ec65; no voters:
I20260806 06:30:47.998847 21840 leader_election.cc:288] T 00000000000000000000000000000000 P 8a8f9156030c47c48add5e8125a7ec65 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 06:30:47.998881 21843 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8a8f9156030c47c48add5e8125a7ec65 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 06:30:47.999023 21843 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8a8f9156030c47c48add5e8125a7ec65 [term 1 LEADER]: Becoming Leader. State: Replica: 8a8f9156030c47c48add5e8125a7ec65, State: Running, Role: LEADER
I20260806 06:30:47.999079 21840 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8a8f9156030c47c48add5e8125a7ec65 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 06:30:47.999114 21843 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8a8f9156030c47c48add5e8125a7ec65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a8f9156030c47c48add5e8125a7ec65" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 39891 } }
I20260806 06:30:47.999447 21844 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a8f9156030c47c48add5e8125a7ec65 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8a8f9156030c47c48add5e8125a7ec65" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a8f9156030c47c48add5e8125a7ec65" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 39891 } } }
I20260806 06:30:47.999452 21845 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a8f9156030c47c48add5e8125a7ec65 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8a8f9156030c47c48add5e8125a7ec65. Latest consensus state: current_term: 1 leader_uuid: "8a8f9156030c47c48add5e8125a7ec65" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a8f9156030c47c48add5e8125a7ec65" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 39891 } } }
I20260806 06:30:47.999547 21844 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a8f9156030c47c48add5e8125a7ec65 [sys.catalog]: This master's current role is: LEADER
I20260806 06:30:47.999550 21845 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a8f9156030c47c48add5e8125a7ec65 [sys.catalog]: This master's current role is: LEADER
I20260806 06:30:47.999769 21851 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 06:30:48.000274 21851 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 06:30:48.001202 21554 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHhLhT/build/release/bin/kudu as pid 21744
I20260806 06:30:48.001308 21554 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785997581049705-21554-0/minicluster-data/master-0/wal/instance
I20260806 06:30:48.001498 21554 external_mini_cluster.cc:1428] Setting key f932b0c106e91e7ab5b3076045d6dae1
I20260806 06:30:48.001748 21851 catalog_manager.cc:1347] Generated new cluster ID: fab64155cabc45ecad306954be32780e
I20260806 06:30:48.001793 21851 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 06:30:48.011902 21866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:48.002109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35491 (local address 127.21.12.190:39891)
0806 06:30:48.002258 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0806 06:30:48.002267 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 06:30:48.010504 (+ 8237us) server_negotiation.cc:407] Connection header received
0806 06:30:48.010634 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:30:48.010641 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:30:48.010665 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:30:48.010785 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:30:48.011490 (+ 705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:30:48.011494 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:30:48.011494 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 06:30:48.011498 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:30:48.011533 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 06:30:48.011545 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:30:48.011598 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0806 06:30:48.011661 (+ 63us) server_negotiation.cc:299] Negotiation successful
0806 06:30:48.011705 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":68,"threads_started":1}
I20260806 06:30:48.012760 21554 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 06:30:48.015026 21866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:48.014046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39659 (local address 127.21.12.190:39891)
0806 06:30:48.014079 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0806 06:30:48.014081 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:30:48.014219 (+ 138us) server_negotiation.cc:407] Connection header received
0806 06:30:48.014275 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:30:48.014276 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:30:48.014281 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:30:48.014345 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:30:48.014843 (+ 498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:30:48.014844 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:30:48.014845 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 06:30:48.014849 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:30:48.014861 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 06:30:48.014865 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:30:48.014906 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0806 06:30:48.014954 (+ 48us) server_negotiation.cc:299] Negotiation successful
0806 06:30:48.014966 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260806 06:30:48.019454 21851 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 06:30:48.019614 21851 catalog_manager.cc:1504] Loading token signing keys...
I20260806 06:30:48.025389 21851 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8a8f9156030c47c48add5e8125a7ec65: Generated new TSK 0
I20260806 06:30:48.025550 21851 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 06:30:48.033412 21554 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHhLhT/build/release/bin/kudu with pid 21744
2026-08-06T06:30:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1383 ms)
[----------] 18 tests from AuthTokenIssuingTest (25129 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785997581049705-21554-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 06:30:48 dist-test-slave-4tdj krb5kdc[21876](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 06:30:48 dist-test-slave-4tdj krb5kdc[21876](info): set up 2 sockets
Aug 06 06:30:48 dist-test-slave-4tdj krb5kdc[21876](info): commencing operation
krb5kdc: starting...
W20260806 06:30:49.146425 21554 mini_kdc.cc:121] Time spent starting KDC: real 1.092s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785997581049705-21554-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785997581049705-21554-0/krb5kdc/test-user.keytab.
Aug 06 06:30:49 dist-test-slave-4tdj krb5kdc[21876](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997849, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T06:30:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T06:30:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.12.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.12.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.12.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.190.keytab.
Entry for principal kudu/127.21.12.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.190.keytab.
I20260806 06:30:49.259322 21554 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHhLhT/build/release/bin/kudu
/tmp/dist-test-taskFHhLhT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785997581049705-21554-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785997581049705-21554-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785997581049705-21554-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785997581049705-21554-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.12.190:33633
--webserver_interface=127.21.12.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.190.keytab
--principal=kudu/127.21.12.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.148:35017
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.12.190:33633,127.21.12.189:43499,127.21.12.188:45467
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785997581049705-21554-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785997581049705-21554-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785997581049705-21554-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 06:30:49.338151 21894 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260806 06:30:49.338398 21894 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 06:30:49.338420 21894 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 06:30:49.338440 21894 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 06:30:49.339553 21894 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 06:30:49.339586 21894 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 06:30:49.339599 21894 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 06:30:49.339612 21894 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 06:30:49.339622 21894 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 06:30:49.341040 21894 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.148:35017
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785997581049705-21554-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785997581049705-21554-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.12.190:33633,127.21.12.189:43499,127.21.12.188:45467
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.190.keytab
--principal=kudu/127.21.12.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.12.190:33633
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785997581049705-21554-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.12.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21894
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785997581049705-21554-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 05:11:02 UTC on ccycloud.quasar-hvzbbz.root.comops.site
build id 4569617
I20260806 06:30:49.341230 21894 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 06:30:49.341405 21894 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 06:30:49.346035 21900 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:30:49.346047 21899 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:30:49.346241 21894 server_base.cc:1034] running on GCE node
W20260806 06:30:49.346134 21904 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:30:49.346560 21894 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 06:30:49.346850 21894 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 06:30:49.347990 21894 hybrid_clock.cc:648] HybridClock initialized: now 1785997849347976 us; error 29 us; skew 500 ppm
Aug 06 06:30:49 dist-test-slave-4tdj krb5kdc[21876](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997849, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 06:30:49.354727 21894 init.cc:377] Logged in from keytab as kudu/127.21.12.190@KRBTEST.COM (short username kudu)
I20260806 06:30:49.355161 21894 webserver.cc:460] Webserver started at http://127.21.12.190:38095/ using document root <none> and password file <none>
I20260806 06:30:49.355381 21894 fs_manager.cc:359] Metadata directory not provided
I20260806 06:30:49.355429 21894 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 06:30:49.355551 21894 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 06:30:49.356551 21894 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785997581049705-21554-0/minicluster-data/master-0/data/instance:
uuid: "527b23341eb44fa3807ee2a8af0b16d7"
format_stamp: "Formatted at 2026-08-06 06:30:49 on dist-test-slave-4tdj"
server_key: "0ed0d1308014da9978548a4e6d93c894"
server_key_iv: "3f128319a5466c7b8d0841568c249528"
server_key_version: "encryptionkey@0"
I20260806 06:30:49.356882 21894 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785997581049705-21554-0/minicluster-data/master-0/wal/instance:
uuid: "527b23341eb44fa3807ee2a8af0b16d7"
format_stamp: "Formatted at 2026-08-06 06:30:49 on dist-test-slave-4tdj"
server_key: "0ed0d1308014da9978548a4e6d93c894"
server_key_iv: "3f128319a5466c7b8d0841568c249528"
server_key_version: "encryptionkey@0"
I20260806 06:30:49.358137 21894 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260806 06:30:49.358866 21914 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 06:30:49.359015 21894 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260806 06:30:49.359100 21894 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785997581049705-21554-0/minicluster-data/master-0/data,/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785997581049705-21554-0/minicluster-data/master-0/wal
uuid: "527b23341eb44fa3807ee2a8af0b16d7"
format_stamp: "Formatted at 2026-08-06 06:30:49 on dist-test-slave-4tdj"
server_key: "0ed0d1308014da9978548a4e6d93c894"
server_key_iv: "3f128319a5466c7b8d0841568c249528"
server_key_version: "encryptionkey@0"
I20260806 06:30:49.359165 21894 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785997581049705-21554-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785997581049705-21554-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785997581049705-21554-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 06:30:49.382550 21894 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 06:30:49.384295 21894 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 06:30:49.384430 21894 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 06:30:49.388286 21894 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.190:33633
I20260806 06:30:49.388322 21985 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.190:33633 every 8 connection(s)
I20260806 06:30:49.388640 21894 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785997581049705-21554-0/minicluster-data/master-0/data/info.pb
I20260806 06:30:49.389603 21986 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 06:30:49.391155 21986 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.12.190" port: 33633 } has no permanent_uuid. Determining permanent_uuid...
I20260806 06:30:49.393826 21554 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHhLhT/build/release/bin/kudu as pid 21894
I20260806 06:30:49.393915 21554 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785997581049705-21554-0/minicluster-data/master-0/wal/instance
I20260806 06:30:49.394101 21554 external_mini_cluster.cc:1428] Setting key 24fafb1aaa3ef0b3527ea06447b9e2be
Aug 06 06:30:49 dist-test-slave-4tdj krb5kdc[21876](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997849, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.190@KRBTEST.COM for kudu/127.21.12.190@KRBTEST.COM
I20260806 06:30:49.399991 21988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:49.391741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:33633 (local address 127.0.0.1:49696)
0806 06:30:49.391904 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0806 06:30:49.391943 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0806 06:30:49.392391 (+ 448us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:30:49.392643 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:30:49.392645 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:30:49.392879 (+ 234us) client_negotiation.cc:189] Negotiated authn=SASL
0806 06:30:49.393026 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:30:49.393029 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:30:49.394043 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:30:49.394044 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:30:49.394792 (+ 748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:30:49.394795 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:30:49.395259 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:30:49.395260 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:30:49.395320 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:30:49.395774 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 06:30:49.395782 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0806 06:30:49.397065 (+ 1283us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 06:30:49.398646 (+ 1581us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:30:49.398652 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:30:49.398654 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:30:49.398833 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:30:49.399032 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:30:49.399033 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:30:49.399033 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:30:49.399121 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:30:49.399443 (+ 322us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 06:30:49.399449 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 06:30:49.399510 (+ 61us) client_negotiation.cc:769] Sending connection context
0806 06:30:49.399588 (+ 78us) client_negotiation.cc:240] Negotiation successful
0806 06:30:49.399762 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"mutex_wait_us":4,"thread_start_us":72,"threads_started":1}
I20260806 06:30:49.399993 21989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:49.391787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49696 (local address 127.21.12.190:33633)
0806 06:30:49.391919 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0806 06:30:49.391928 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 06:30:49.392020 (+ 92us) server_negotiation.cc:407] Connection header received
0806 06:30:49.392526 (+ 506us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:30:49.392528 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:30:49.392544 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:30:49.392629 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:30:49.393092 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:49.393965 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:49.394874 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:49.395169 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:49.395264 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:30:49.397147 (+ 1883us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:30:49.397154 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:30:49.397154 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 06:30:49.397161 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:30:49.398568 (+ 1407us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:30:49.398900 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:30:49.398903 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:30:49.398904 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:30:49.398977 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:30:49.399175 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:30:49.399176 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:30:49.399176 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:30:49.399377 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:30:49.399453 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0806 06:30:49.399636 (+ 183us) server_negotiation.cc:299] Negotiation successful
0806 06:30:49.399819 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":105,"server-negotiator.queue_time_us":100,"thread_start_us":65,"threads_started":1}
I20260806 06:30:49.400632 21986 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.12.189" port: 43499 } has no permanent_uuid. Determining permanent_uuid...
I20260806 06:30:49.400934 21988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:49.400828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.189:43499 (local address 127.0.0.1:43980)
0806 06:30:49.400867 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0806 06:30:49.400910 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.189:43499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260806 06:30:49.401054 21916 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.12.189:43499: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.21.12.189@KRBTEST.COM; defaulting to no policy
I20260806 06:30:49.401448 21988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:49.401363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.189:43499 (local address 127.0.0.1:43988)
0806 06:30:49.401401 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0806 06:30:49.401425 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.189:43499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260806 06:30:49.401535 21986 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.12.189:43499: Network error: Client connection negotiation failed: client connection to 127.21.12.189:43499: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.12.189@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.12.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.189.keytab.
Entry for principal kudu/127.21.12.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.189.keytab.
I20260806 06:30:49.416813 21554 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHhLhT/build/release/bin/kudu
/tmp/dist-test-taskFHhLhT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785997581049705-21554-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785997581049705-21554-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785997581049705-21554-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785997581049705-21554-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.12.189:43499
--webserver_interface=127.21.12.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.189.keytab
--principal=kudu/127.21.12.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.148:35017
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.12.190:33633,127.21.12.189:43499,127.21.12.188:45467
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785997581049705-21554-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785997581049705-21554-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785997581049705-21554-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 06:30:49.450730 21986 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.12.189" port: 43499 } attempt: 1
I20260806 06:30:49.451181 21988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:49.451061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.189:43499 (local address 127.0.0.1:43996)
0806 06:30:49.451115 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0806 06:30:49.451152 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.189:43499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260806 06:30:49.451519 21988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:49.451436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.189:43499 (local address 127.0.0.1:44010)
0806 06:30:49.451475 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0806 06:30:49.451500 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.189:43499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260806 06:30:49.451615 21986 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.12.189:43499: Network error: Client connection negotiation failed: client connection to 127.21.12.189:43499: connect: Connection refused (error 111)
W20260806 06:30:49.495478 21993 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260806 06:30:49.495694 21993 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 06:30:49.495719 21993 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 06:30:49.495743 21993 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 06:30:49.496830 21993 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 06:30:49.496868 21993 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 06:30:49.496886 21993 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 06:30:49.496902 21993 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 06:30:49.496917 21993 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 06:30:49.498383 21993 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.148:35017
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785997581049705-21554-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785997581049705-21554-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.21.12.190:33633,127.21.12.189:43499,127.21.12.188:45467
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.189.keytab
--principal=kudu/127.21.12.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.12.189:43499
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785997581049705-21554-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.21.12.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21993
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785997581049705-21554-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 05:11:02 UTC on ccycloud.quasar-hvzbbz.root.comops.site
build id 4569617
I20260806 06:30:49.498620 21993 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 06:30:49.498814 21993 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260806 06:30:49.504447 21993 server_base.cc:1034] running on GCE node
W20260806 06:30:49.504412 22000 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:30:49.504441 22002 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:30:49.504407 21998 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:30:49.504819 21993 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 06:30:49.505093 21993 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 06:30:49.506232 21993 hybrid_clock.cc:648] HybridClock initialized: now 1785997849506217 us; error 27 us; skew 500 ppm
Aug 06 06:30:49 dist-test-slave-4tdj krb5kdc[21876](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997849, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 06:30:49.508513 21993 init.cc:377] Logged in from keytab as kudu/127.21.12.189@KRBTEST.COM (short username kudu)
I20260806 06:30:49.508936 21993 webserver.cc:460] Webserver started at http://127.21.12.189:44751/ using document root <none> and password file <none>
I20260806 06:30:49.509124 21993 fs_manager.cc:359] Metadata directory not provided
I20260806 06:30:49.509171 21993 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 06:30:49.509290 21993 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 06:30:49.510317 21993 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785997581049705-21554-0/minicluster-data/master-1/data/instance:
uuid: "8b04b33b01dc4d07a4aa7b5a7470013b"
format_stamp: "Formatted at 2026-08-06 06:30:49 on dist-test-slave-4tdj"
server_key: "15045f2e204e0b17df7af8e8ff8827a5"
server_key_iv: "64b924cd29da1a45dcbf339ed3c69470"
server_key_version: "encryptionkey@0"
I20260806 06:30:49.510671 21993 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785997581049705-21554-0/minicluster-data/master-1/wal/instance:
uuid: "8b04b33b01dc4d07a4aa7b5a7470013b"
format_stamp: "Formatted at 2026-08-06 06:30:49 on dist-test-slave-4tdj"
server_key: "15045f2e204e0b17df7af8e8ff8827a5"
server_key_iv: "64b924cd29da1a45dcbf339ed3c69470"
server_key_version: "encryptionkey@0"
I20260806 06:30:49.511861 21993 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260806 06:30:49.512503 22012 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 06:30:49.512650 21993 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260806 06:30:49.512740 21993 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785997581049705-21554-0/minicluster-data/master-1/data,/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785997581049705-21554-0/minicluster-data/master-1/wal
uuid: "8b04b33b01dc4d07a4aa7b5a7470013b"
format_stamp: "Formatted at 2026-08-06 06:30:49 on dist-test-slave-4tdj"
server_key: "15045f2e204e0b17df7af8e8ff8827a5"
server_key_iv: "64b924cd29da1a45dcbf339ed3c69470"
server_key_version: "encryptionkey@0"
I20260806 06:30:49.512804 21993 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785997581049705-21554-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785997581049705-21554-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785997581049705-21554-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 06:30:49.519747 21986 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.12.189" port: 43499 } attempt: 2
I20260806 06:30:49.520069 21988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:49.519965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.189:43499 (local address 127.0.0.1:44018)
0806 06:30:49.520014 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0806 06:30:49.520042 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.189:43499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260806 06:30:49.520357 21988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:49.520281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.189:43499 (local address 127.0.0.1:44028)
0806 06:30:49.520316 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0806 06:30:49.520340 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.189:43499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
W20260806 06:30:49.520434 21986 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.12.189:43499: Network error: Client connection negotiation failed: client connection to 127.21.12.189:43499: connect: Connection refused (error 111)
I20260806 06:30:49.524667 21993 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 06:30:49.526368 21993 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 06:30:49.526508 21993 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 06:30:49.530342 21993 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.189:43499
I20260806 06:30:49.530387 22081 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.189:43499 every 8 connection(s)
I20260806 06:30:49.530700 21993 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785997581049705-21554-0/minicluster-data/master-1/data/info.pb
I20260806 06:30:49.531286 21554 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHhLhT/build/release/bin/kudu as pid 21993
I20260806 06:30:49.531401 21554 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785997581049705-21554-0/minicluster-data/master-1/wal/instance
I20260806 06:30:49.531631 21554 external_mini_cluster.cc:1428] Setting key 3f2e75040a64213df550d2c2d5a20d8f
I20260806 06:30:49.531863 22082 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 06:30:49.533762 22082 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.12.190" port: 33633 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.21.12.188@KRBTEST.COM; defaulting to no policy
Aug 06 06:30:49 dist-test-slave-4tdj krb5kdc[21876](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997849, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.189@KRBTEST.COM for kudu/127.21.12.190@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.12.188@KRBTEST.COM" created.
I20260806 06:30:49.541852 21989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:49.534388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49706 (local address 127.21.12.190:33633)
0806 06:30:49.534439 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0806 06:30:49.534441 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:30:49.534656 (+ 215us) server_negotiation.cc:407] Connection header received
0806 06:30:49.535119 (+ 463us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:30:49.535120 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:30:49.535133 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:30:49.535183 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:30:49.535801 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:49.536283 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:49.537203 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:49.537413 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:49.537467 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:30:49.539482 (+ 2015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:30:49.539483 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:30:49.539484 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 06:30:49.539492 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:30:49.540624 (+ 1132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:30:49.541016 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:30:49.541017 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:30:49.541018 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:30:49.541075 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:30:49.541266 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:30:49.541267 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:30:49.541267 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:30:49.541361 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:30:49.541412 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0806 06:30:49.541718 (+ 306us) server_negotiation.cc:299] Negotiation successful
0806 06:30:49.541788 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260806 06:30:49.542064 22085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:49.534374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:33633 (local address 127.0.0.1:49706)
0806 06:30:49.534538 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0806 06:30:49.534577 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0806 06:30:49.535029 (+ 452us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:30:49.535247 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:30:49.535251 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:30:49.535531 (+ 280us) client_negotiation.cc:189] Negotiated authn=SASL
0806 06:30:49.535715 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:30:49.535719 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:30:49.536364 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:30:49.536365 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:30:49.537123 (+ 758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:30:49.537125 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:30:49.537507 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:30:49.537509 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:30:49.537610 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:30:49.538040 (+ 430us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 06:30:49.538056 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 06:30:49.539389 (+ 1333us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 06:30:49.540694 (+ 1305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:30:49.540702 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:30:49.540703 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:30:49.540939 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:30:49.541125 (+ 186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:30:49.541125 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:30:49.541125 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:30:49.541224 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:30:49.541409 (+ 185us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 06:30:49.541414 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 06:30:49.541550 (+ 136us) client_negotiation.cc:769] Sending connection context
0806 06:30:49.541690 (+ 140us) client_negotiation.cc:240] Negotiation successful
0806 06:30:49.541846 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":84,"threads_started":1}
I20260806 06:30:49.542641 22082 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.12.189" port: 43499 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 06:30:49 dist-test-slave-4tdj krb5kdc[21876](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997849, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.189@KRBTEST.COM for kudu/127.21.12.189@KRBTEST.COM
I20260806 06:30:49.549181 22085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:49.542859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.189:43499 (local address 127.0.0.1:44036)
0806 06:30:49.542895 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0806 06:30:49.542905 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0806 06:30:49.542941 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:30:49.543247 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:30:49.543247 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:30:49.543348 (+ 101us) client_negotiation.cc:189] Negotiated authn=SASL
0806 06:30:49.543393 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:30:49.543394 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:30:49.544211 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:30:49.544212 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:30:49.544674 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:30:49.544676 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:30:49.545113 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:30:49.545114 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:30:49.545174 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:30:49.545537 (+ 363us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 06:30:49.545542 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0806 06:30:49.546781 (+ 1239us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 06:30:49.548280 (+ 1499us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:30:49.548281 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:30:49.548281 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:30:49.548426 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:30:49.548599 (+ 173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:30:49.548600 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:30:49.548600 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:30:49.548651 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:30:49.548888 (+ 237us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 06:30:49.548888 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 06:30:49.548925 (+ 37us) client_negotiation.cc:769] Sending connection context
0806 06:30:49.548984 (+ 59us) client_negotiation.cc:240] Negotiation successful
0806 06:30:49.549074 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22}
I20260806 06:30:49.549253 22088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:49.542921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44036 (local address 127.21.12.189:43499)
0806 06:30:49.543063 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0806 06:30:49.543068 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 06:30:49.543091 (+ 23us) server_negotiation.cc:407] Connection header received
0806 06:30:49.543114 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:30:49.543116 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:30:49.543131 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:30:49.543202 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:30:49.543459 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:49.544131 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:49.544739 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:49.545052 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:49.545108 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:30:49.546855 (+ 1747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:30:49.546863 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:30:49.546863 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 06:30:49.546869 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:30:49.548208 (+ 1339us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:30:49.548490 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:30:49.548493 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:30:49.548493 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:30:49.548547 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:30:49.548699 (+ 152us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:30:49.548700 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:30:49.548700 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:30:49.548829 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:30:49.548899 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0806 06:30:49.549012 (+ 113us) server_negotiation.cc:299] Negotiation successful
0806 06:30:49.549142 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":49,"server-negotiator.queue_time_us":112,"thread_start_us":74,"threads_started":1}
I20260806 06:30:49.549945 22082 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.12.188" port: 45467 } has no permanent_uuid. Determining permanent_uuid...
I20260806 06:30:49.550256 22085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:49.550158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.188:45467 (local address 127.0.0.1:55106)
0806 06:30:49.550197 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0806 06:30:49.550239 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.188:45467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260806 06:30:49.550381 22017 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.12.188:45467: connect: Connection refused (error 111)
I20260806 06:30:49.550794 22085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:49.550708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.188:45467 (local address 127.0.0.1:55108)
0806 06:30:49.550745 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0806 06:30:49.550773 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.188:45467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260806 06:30:49.550887 22082 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.12.188:45467: Network error: Client connection negotiation failed: client connection to 127.21.12.188:45467: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.12.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.188.keytab.
Entry for principal kudu/127.21.12.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.188.keytab.
I20260806 06:30:49.556295 21554 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHhLhT/build/release/bin/kudu
/tmp/dist-test-taskFHhLhT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785997581049705-21554-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785997581049705-21554-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785997581049705-21554-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785997581049705-21554-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.12.188:45467
--webserver_interface=127.21.12.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.188.keytab
--principal=kudu/127.21.12.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.148:35017
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.12.190:33633,127.21.12.189:43499,127.21.12.188:45467
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785997581049705-21554-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785997581049705-21554-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785997581049705-21554-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 06:30:49.600085 22082 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.12.188" port: 45467 } attempt: 1
I20260806 06:30:49.600538 22085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:49.600419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.188:45467 (local address 127.0.0.1:55120)
0806 06:30:49.600473 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0806 06:30:49.600506 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.188:45467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260806 06:30:49.600931 22085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:49.600844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.188:45467 (local address 127.0.0.1:55124)
0806 06:30:49.600887 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0806 06:30:49.600912 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.188:45467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260806 06:30:49.601009 22082 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.12.188:45467: Network error: Client connection negotiation failed: client connection to 127.21.12.188:45467: connect: Connection refused (error 111)
I20260806 06:30:49.611570 21986 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.12.189" port: 43499 } attempt: 3
Aug 06 06:30:49 dist-test-slave-4tdj krb5kdc[21876](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997849, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.190@KRBTEST.COM for kudu/127.21.12.189@KRBTEST.COM
I20260806 06:30:49.618517 22088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:49.611939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44050 (local address 127.21.12.189:43499)
0806 06:30:49.612012 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0806 06:30:49.612014 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:30:49.612027 (+ 13us) server_negotiation.cc:407] Connection header received
0806 06:30:49.612081 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:30:49.612082 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:30:49.612096 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:30:49.612142 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:30:49.612442 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:49.613215 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:49.613896 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:49.614261 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:49.614365 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:30:49.616194 (+ 1829us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:30:49.616196 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:30:49.616196 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 06:30:49.616204 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:30:49.617427 (+ 1223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:30:49.617824 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:30:49.617825 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:30:49.617826 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:30:49.617889 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:30:49.618076 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:30:49.618077 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:30:49.618077 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:30:49.618179 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:30:49.618228 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0806 06:30:49.618388 (+ 160us) server_negotiation.cc:299] Negotiation successful
0806 06:30:49.618456 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260806 06:30:49.618569 21988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:49.611888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.189:43499 (local address 127.0.0.1:44050)
0806 06:30:49.611956 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0806 06:30:49.611968 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 06:30:49.612022 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:30:49.612167 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:30:49.612168 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:30:49.612315 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
0806 06:30:49.612382 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:30:49.612384 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:30:49.613283 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:30:49.613284 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:30:49.613815 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:30:49.613817 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:30:49.614368 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:30:49.614369 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:30:49.614424 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:30:49.614825 (+ 401us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 06:30:49.614830 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0806 06:30:49.616085 (+ 1255us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 06:30:49.617507 (+ 1422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:30:49.617509 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:30:49.617510 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:30:49.617726 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:30:49.617955 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:30:49.617955 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:30:49.617956 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:30:49.618025 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:30:49.618228 (+ 203us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 06:30:49.618229 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 06:30:49.618280 (+ 51us) client_negotiation.cc:769] Sending connection context
0806 06:30:49.618370 (+ 90us) client_negotiation.cc:240] Negotiation successful
0806 06:30:49.618470 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260806 06:30:49.619021 21986 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.12.188" port: 45467 } has no permanent_uuid. Determining permanent_uuid...
I20260806 06:30:49.619336 21988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:49.619238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.188:45467 (local address 127.0.0.1:55138)
0806 06:30:49.619281 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0806 06:30:49.619313 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.188:45467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260806 06:30:49.619570 21988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:49.619510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.188:45467 (local address 127.0.0.1:55148)
0806 06:30:49.619532 (+ 22us) negotiation.cc:107] Waiting for socket to connect
0806 06:30:49.619555 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.188:45467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":13}
W20260806 06:30:49.619643 21986 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.12.188:45467: Network error: Client connection negotiation failed: client connection to 127.21.12.188:45467: connect: Connection refused (error 111)
W20260806 06:30:49.639310 22091 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260806 06:30:49.639511 22091 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 06:30:49.639528 22091 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 06:30:49.639560 22091 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 06:30:49.640659 22091 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 06:30:49.640695 22091 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 06:30:49.640708 22091 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 06:30:49.640733 22091 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 06:30:49.640753 22091 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 06:30:49.642198 22091 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.148:35017
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785997581049705-21554-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785997581049705-21554-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.21.12.190:33633,127.21.12.189:43499,127.21.12.188:45467
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.188.keytab
--principal=kudu/127.21.12.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.12.188:45467
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785997581049705-21554-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.21.12.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22091
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785997581049705-21554-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 05:11:02 UTC on ccycloud.quasar-hvzbbz.root.comops.site
build id 4569617
I20260806 06:30:49.642426 22091 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 06:30:49.642642 22091 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 06:30:49.647075 22099 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:30:49.647105 22098 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:30:49.647101 22104 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:30:49.647264 22091 server_base.cc:1034] running on GCE node
I20260806 06:30:49.647545 22091 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 06:30:49.647822 22091 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 06:30:49.648968 22091 hybrid_clock.cc:648] HybridClock initialized: now 1785997849648953 us; error 29 us; skew 500 ppm
I20260806 06:30:49.650779 21986 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.12.188" port: 45467 } attempt: 1
Aug 06 06:30:49 dist-test-slave-4tdj krb5kdc[21876](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997849, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 06:30:49.651160 21988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:49.651015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.188:45467 (local address 127.0.0.1:55164)
0806 06:30:49.651079 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0806 06:30:49.651122 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.188:45467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260806 06:30:49.651499 21988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:49.651417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.188:45467 (local address 127.0.0.1:55174)
0806 06:30:49.651454 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0806 06:30:49.651479 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.188:45467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260806 06:30:49.651482 22091 init.cc:377] Logged in from keytab as kudu/127.21.12.188@KRBTEST.COM (short username kudu)
W20260806 06:30:49.651598 21986 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.12.188:45467: Network error: Client connection negotiation failed: client connection to 127.21.12.188:45467: connect: Connection refused (error 111)
I20260806 06:30:49.651960 22091 webserver.cc:460] Webserver started at http://127.21.12.188:37703/ using document root <none> and password file <none>
I20260806 06:30:49.652196 22091 fs_manager.cc:359] Metadata directory not provided
I20260806 06:30:49.652241 22091 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 06:30:49.652366 22091 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 06:30:49.653407 22091 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785997581049705-21554-0/minicluster-data/master-2/data/instance:
uuid: "b6869d74fc9d4af4b76a0146a1c8e1db"
format_stamp: "Formatted at 2026-08-06 06:30:49 on dist-test-slave-4tdj"
server_key: "dcbd9114a83264f6c38b477261791ac6"
server_key_iv: "df7674cc8e2e89486256af4b3539934b"
server_key_version: "encryptionkey@0"
I20260806 06:30:49.653738 22091 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785997581049705-21554-0/minicluster-data/master-2/wal/instance:
uuid: "b6869d74fc9d4af4b76a0146a1c8e1db"
format_stamp: "Formatted at 2026-08-06 06:30:49 on dist-test-slave-4tdj"
server_key: "dcbd9114a83264f6c38b477261791ac6"
server_key_iv: "df7674cc8e2e89486256af4b3539934b"
server_key_version: "encryptionkey@0"
I20260806 06:30:49.655031 22091 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260806 06:30:49.655700 22111 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 06:30:49.655859 22091 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260806 06:30:49.655952 22091 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785997581049705-21554-0/minicluster-data/master-2/data,/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785997581049705-21554-0/minicluster-data/master-2/wal
uuid: "b6869d74fc9d4af4b76a0146a1c8e1db"
format_stamp: "Formatted at 2026-08-06 06:30:49 on dist-test-slave-4tdj"
server_key: "dcbd9114a83264f6c38b477261791ac6"
server_key_iv: "df7674cc8e2e89486256af4b3539934b"
server_key_version: "encryptionkey@0"
I20260806 06:30:49.656024 22091 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785997581049705-21554-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785997581049705-21554-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785997581049705-21554-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 06:30:49.669140 22082 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.12.188" port: 45467 } attempt: 2
I20260806 06:30:49.669528 22085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:49.669415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.188:45467 (local address 127.0.0.1:55180)
0806 06:30:49.669470 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0806 06:30:49.669499 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.188:45467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260806 06:30:49.669873 22085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:49.669786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.188:45467 (local address 127.0.0.1:55182)
0806 06:30:49.669826 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0806 06:30:49.669851 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.188:45467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260806 06:30:49.669962 22082 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.12.188:45467: Network error: Client connection negotiation failed: client connection to 127.21.12.188:45467: connect: Connection refused (error 111)
I20260806 06:30:49.677286 22091 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 06:30:49.679061 22091 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 06:30:49.679201 22091 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 06:30:49.682953 22091 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.188:45467
I20260806 06:30:49.682986 22177 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.188:45467 every 8 connection(s)
I20260806 06:30:49.683307 22091 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785997581049705-21554-0/minicluster-data/master-2/data/info.pb
I20260806 06:30:49.684260 22178 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 06:30:49.685788 22178 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.12.190" port: 33633 } has no permanent_uuid. Determining permanent_uuid...
I20260806 06:30:49.690843 21554 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHhLhT/build/release/bin/kudu as pid 22091
I20260806 06:30:49.690946 21554 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785997581049705-21554-0/minicluster-data/master-2/wal/instance
I20260806 06:30:49.691141 21554 external_mini_cluster.cc:1428] Setting key f697bb3e82184edce9a16d584b5330ec
Aug 06 06:30:49 dist-test-slave-4tdj krb5kdc[21876](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997849, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.188@KRBTEST.COM for kudu/127.21.12.190@KRBTEST.COM
I20260806 06:30:49.694439 21989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:49.686475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49714 (local address 127.21.12.190:33633)
0806 06:30:49.686521 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0806 06:30:49.686523 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:30:49.686759 (+ 236us) server_negotiation.cc:407] Connection header received
0806 06:30:49.687255 (+ 496us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:30:49.687262 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:30:49.687275 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:30:49.687327 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:30:49.687916 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:49.688378 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:49.689298 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:49.689481 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:49.689537 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:30:49.691769 (+ 2232us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:30:49.691770 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:30:49.691771 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 06:30:49.691778 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:30:49.692991 (+ 1213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:30:49.693412 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:30:49.693413 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:30:49.693413 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:30:49.693473 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:30:49.693696 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:30:49.693697 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:30:49.693697 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:30:49.693829 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:30:49.693895 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0806 06:30:49.694175 (+ 280us) server_negotiation.cc:299] Negotiation successful
0806 06:30:49.694366 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260806 06:30:49.694630 22180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:49.686459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:33633 (local address 127.0.0.1:49714)
0806 06:30:49.686643 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0806 06:30:49.686681 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0806 06:30:49.687145 (+ 464us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:30:49.687402 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:30:49.687406 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:30:49.687644 (+ 238us) client_negotiation.cc:189] Negotiated authn=SASL
0806 06:30:49.687829 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:30:49.687833 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:30:49.688454 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:30:49.688455 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:30:49.689219 (+ 764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:30:49.689221 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:30:49.689570 (+ 349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:30:49.689571 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:30:49.689676 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:30:49.690079 (+ 403us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 06:30:49.690086 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0806 06:30:49.691673 (+ 1587us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 06:30:49.693063 (+ 1390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:30:49.693070 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:30:49.693072 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:30:49.693347 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:30:49.693531 (+ 184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:30:49.693531 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:30:49.693532 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:30:49.693643 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:30:49.693916 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 06:30:49.693922 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 06:30:49.694043 (+ 121us) client_negotiation.cc:769] Sending connection context
0806 06:30:49.694146 (+ 103us) client_negotiation.cc:240] Negotiation successful
0806 06:30:49.694327 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":94,"threads_started":1}
I20260806 06:30:49.695190 22178 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.12.189" port: 43499 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 06:30:49 dist-test-slave-4tdj krb5kdc[21876](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997849, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.12.188@KRBTEST.COM
Aug 06 06:30:49 dist-test-slave-4tdj krb5kdc[21876](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997849, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.188@KRBTEST.COM for kudu/127.21.12.189@KRBTEST.COM
I20260806 06:30:49.699716 22182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:49.691508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55190 (local address 127.21.12.188:45467)
0806 06:30:49.691660 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0806 06:30:49.691665 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 06:30:49.691685 (+ 20us) server_negotiation.cc:407] Connection header received
0806 06:30:49.691709 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:30:49.691711 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:30:49.691724 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:30:49.691796 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:30:49.692499 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:49.693269 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:49.694169 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:49.694612 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:49.694683 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:30:49.697182 (+ 2499us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:30:49.697191 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:30:49.697191 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 06:30:49.697198 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:30:49.698644 (+ 1446us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:30:49.698961 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:30:49.698964 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:30:49.698965 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:30:49.699023 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:30:49.699194 (+ 171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:30:49.699195 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:30:49.699195 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:30:49.699330 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:30:49.699464 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0806 06:30:49.699537 (+ 73us) server_negotiation.cc:299] Negotiation successful
0806 06:30:49.699619 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":81,"threads_started":1}
I20260806 06:30:49.701465 22180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:49.695409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.189:43499 (local address 127.0.0.1:44060)
0806 06:30:49.695448 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0806 06:30:49.695458 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0806 06:30:49.695491 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:30:49.695677 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:30:49.695678 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:30:49.695785 (+ 107us) client_negotiation.cc:189] Negotiated authn=SASL
0806 06:30:49.695827 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:30:49.695828 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:30:49.696541 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:30:49.696542 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:30:49.697115 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:30:49.697118 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:30:49.697472 (+ 354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:30:49.697473 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:30:49.697546 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:30:49.697833 (+ 287us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 06:30:49.697837 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0806 06:30:49.698916 (+ 1079us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 06:30:49.700440 (+ 1524us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:30:49.700441 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:30:49.700441 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:30:49.700685 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:30:49.700865 (+ 180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:30:49.700866 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:30:49.700866 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:30:49.700921 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:30:49.701213 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 06:30:49.701214 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 06:30:49.701246 (+ 32us) client_negotiation.cc:769] Sending connection context
0806 06:30:49.701296 (+ 50us) client_negotiation.cc:240] Negotiation successful
0806 06:30:49.701379 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24}
I20260806 06:30:49.701486 22088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:49.695485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44060 (local address 127.21.12.189:43499)
0806 06:30:49.695527 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0806 06:30:49.695530 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 06:30:49.695538 (+ 8us) server_negotiation.cc:407] Connection header received
0806 06:30:49.695562 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:30:49.695564 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:30:49.695605 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:30:49.695676 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:30:49.695889 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:49.696451 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:49.697200 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:49.697406 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:49.697463 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:30:49.698990 (+ 1527us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:30:49.698991 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:30:49.698991 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 06:30:49.698999 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:30:49.700352 (+ 1353us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:30:49.700751 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:30:49.700752 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:30:49.700752 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:30:49.700813 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:30:49.700995 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:30:49.700996 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:30:49.700996 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:30:49.701133 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:30:49.701174 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0806 06:30:49.701336 (+ 162us) server_negotiation.cc:299] Negotiation successful
0806 06:30:49.701403 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260806 06:30:49.702203 22178 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.12.188" port: 45467 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 06:30:49 dist-test-slave-4tdj krb5kdc[21876](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997849, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.12.190@KRBTEST.COM
Aug 06 06:30:49 dist-test-slave-4tdj krb5kdc[21876](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997849, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.188@KRBTEST.COM for kudu/127.21.12.188@KRBTEST.COM
I20260806 06:30:49.707015 21989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:49.700424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49716 (local address 127.21.12.190:33633)
0806 06:30:49.700464 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0806 06:30:49.700466 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:30:49.700473 (+ 7us) server_negotiation.cc:407] Connection header received
0806 06:30:49.700526 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:30:49.700527 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:30:49.700536 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:30:49.700606 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:30:49.701021 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:49.701454 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:49.702060 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:49.702324 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:49.702386 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:30:49.704709 (+ 2323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:30:49.704710 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:30:49.704711 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 06:30:49.704717 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:30:49.705948 (+ 1231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:30:49.706313 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:30:49.706314 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:30:49.706314 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:30:49.706376 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:30:49.706554 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:30:49.706555 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:30:49.706555 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:30:49.706660 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:30:49.706812 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0806 06:30:49.706852 (+ 40us) server_negotiation.cc:299] Negotiation successful
0806 06:30:49.706926 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260806 06:30:49.708947 22180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:49.702469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.188:45467 (local address 127.0.0.1:55198)
0806 06:30:49.702599 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0806 06:30:49.702608 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0806 06:30:49.702649 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:30:49.702772 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:30:49.702773 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:30:49.702874 (+ 101us) client_negotiation.cc:189] Negotiated authn=SASL
0806 06:30:49.702901 (+ 27us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:30:49.702902 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:30:49.703744 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:30:49.703745 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:30:49.704470 (+ 725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:30:49.704473 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:30:49.704805 (+ 332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:30:49.704806 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:30:49.704852 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:30:49.705225 (+ 373us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 06:30:49.705229 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0806 06:30:49.706455 (+ 1226us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 06:30:49.708058 (+ 1603us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:30:49.708059 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:30:49.708060 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:30:49.708247 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:30:49.708422 (+ 175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:30:49.708422 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:30:49.708422 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:30:49.708465 (+ 43us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:30:49.708660 (+ 195us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 06:30:49.708660 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 06:30:49.708691 (+ 31us) client_negotiation.cc:769] Sending connection context
0806 06:30:49.708735 (+ 44us) client_negotiation.cc:240] Negotiation successful
0806 06:30:49.708809 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116}
I20260806 06:30:49.708992 22182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:49.702509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55198 (local address 127.21.12.188:45467)
0806 06:30:49.702543 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0806 06:30:49.702544 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0806 06:30:49.702649 (+ 105us) server_negotiation.cc:407] Connection header received
0806 06:30:49.702713 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:30:49.702714 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:30:49.702722 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:30:49.702763 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:30:49.702963 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:49.703672 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:49.704542 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:49.704738 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:49.704797 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:30:49.706523 (+ 1726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:30:49.706524 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:30:49.706524 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 06:30:49.706530 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:30:49.707989 (+ 1459us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:30:49.708305 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:30:49.708306 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:30:49.708306 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:30:49.708363 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:30:49.708504 (+ 141us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:30:49.708504 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:30:49.708504 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:30:49.708613 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:30:49.708666 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0806 06:30:49.708754 (+ 88us) server_negotiation.cc:299] Negotiation successful
0806 06:30:49.708866 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":36,"server-negotiator.queue_time_us":17}
I20260806 06:30:49.710522 22178 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6869d74fc9d4af4b76a0146a1c8e1db: Bootstrap starting.
I20260806 06:30:49.711421 22178 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b6869d74fc9d4af4b76a0146a1c8e1db: Neither blocks nor log segments found. Creating new log.
Aug 06 06:30:49 dist-test-slave-4tdj krb5kdc[21876](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997849, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.12.189@KRBTEST.COM
I20260806 06:30:49.711743 22178 log.cc:824] T 00000000000000000000000000000000 P b6869d74fc9d4af4b76a0146a1c8e1db: Log is configured to *not* fsync() on all Append() calls
I20260806 06:30:49.712527 22178 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6869d74fc9d4af4b76a0146a1c8e1db: No bootstrap required, opened a new log
I20260806 06:30:49.714408 22088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:49.707597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44074 (local address 127.21.12.189:43499)
0806 06:30:49.707631 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0806 06:30:49.707633 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:30:49.707657 (+ 24us) server_negotiation.cc:407] Connection header received
0806 06:30:49.707677 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:30:49.707678 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:30:49.707687 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:30:49.707740 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:30:49.708142 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:49.708752 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:49.709576 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:49.709828 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:49.709895 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:30:49.712014 (+ 2119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:30:49.712015 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:30:49.712016 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 06:30:49.712022 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:30:49.713393 (+ 1371us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:30:49.713686 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:30:49.713687 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:30:49.713687 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:30:49.713747 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:30:49.713898 (+ 151us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:30:49.713899 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:30:49.713899 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:30:49.714019 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:30:49.714187 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0806 06:30:49.714224 (+ 37us) server_negotiation.cc:299] Negotiation successful
0806 06:30:49.714321 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260806 06:30:49.714532 22178 raft_consensus.cc:348] T 00000000000000000000000000000000 P b6869d74fc9d4af4b76a0146a1c8e1db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "527b23341eb44fa3807ee2a8af0b16d7" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 33633 } } peers { permanent_uuid: "8b04b33b01dc4d07a4aa7b5a7470013b" member_type: VOTER last_known_addr { host: "127.21.12.189" port: 43499 } } peers { permanent_uuid: "b6869d74fc9d4af4b76a0146a1c8e1db" member_type: VOTER last_known_addr { host: "127.21.12.188" port: 45467 } }
I20260806 06:30:49.714686 22178 raft_consensus.cc:729] T 00000000000000000000000000000000 P b6869d74fc9d4af4b76a0146a1c8e1db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6869d74fc9d4af4b76a0146a1c8e1db, State: Initialized, Role: FOLLOWER
I20260806 06:30:49.714781 22178 consensus_queue.cc:260] T 00000000000000000000000000000000 P b6869d74fc9d4af4b76a0146a1c8e1db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "527b23341eb44fa3807ee2a8af0b16d7" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 33633 } } peers { permanent_uuid: "8b04b33b01dc4d07a4aa7b5a7470013b" member_type: VOTER last_known_addr { host: "127.21.12.189" port: 43499 } } peers { permanent_uuid: "b6869d74fc9d4af4b76a0146a1c8e1db" member_type: VOTER last_known_addr { host: "127.21.12.188" port: 45467 } }
I20260806 06:30:49.715111 22178 sys_catalog.cc:564] T 00000000000000000000000000000000 P b6869d74fc9d4af4b76a0146a1c8e1db [sys.catalog]: configured and running, proceeding with master startup.
I20260806 06:30:49.715068 22184 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6869d74fc9d4af4b76a0146a1c8e1db [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "527b23341eb44fa3807ee2a8af0b16d7" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 33633 } } peers { permanent_uuid: "8b04b33b01dc4d07a4aa7b5a7470013b" member_type: VOTER last_known_addr { host: "127.21.12.189" port: 43499 } } peers { permanent_uuid: "b6869d74fc9d4af4b76a0146a1c8e1db" member_type: VOTER last_known_addr { host: "127.21.12.188" port: 45467 } } }
I20260806 06:30:49.715181 22184 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6869d74fc9d4af4b76a0146a1c8e1db [sys.catalog]: This master's current role is: FOLLOWER
I20260806 06:30:49.715837 21554 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260806 06:30:49.716821 22199 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b6869d74fc9d4af4b76a0146a1c8e1db: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 06:30:49.716877 22199 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 06:30:49.717053 21953 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "527b23341eb44fa3807ee2a8af0b16d7"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49716
I20260806 06:30:49.717097 22091 master_runner.cc:186] Error getting master registration for 127.21.12.190:33633: OK, instance_id { permanent_uuid: "527b23341eb44fa3807ee2a8af0b16d7" instance_seqno: 1785997849387259 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260806 06:30:49.719177 22091 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260806 06:30:49.726694 21986 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.12.188" port: 45467 } attempt: 2
Aug 06 06:30:49 dist-test-slave-4tdj krb5kdc[21876](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997849, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.190@KRBTEST.COM for kudu/127.21.12.188@KRBTEST.COM
I20260806 06:30:49.733806 22182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:49.727015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55204 (local address 127.21.12.188:45467)
0806 06:30:49.727054 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0806 06:30:49.727056 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:30:49.727078 (+ 22us) server_negotiation.cc:407] Connection header received
0806 06:30:49.727144 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:30:49.727145 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:30:49.727153 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:30:49.727208 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:30:49.727493 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:49.728214 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:49.729166 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:49.729452 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:49.729503 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:30:49.731550 (+ 2047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:30:49.731552 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:30:49.731552 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 06:30:49.731560 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:30:49.732810 (+ 1250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:30:49.733112 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:30:49.733113 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:30:49.733113 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:30:49.733165 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:30:49.733323 (+ 158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:30:49.733323 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:30:49.733324 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:30:49.733455 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:30:49.733511 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0806 06:30:49.733650 (+ 139us) server_negotiation.cc:299] Negotiation successful
0806 06:30:49.733718 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260806 06:30:49.733806 21988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:49.726972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.188:45467 (local address 127.0.0.1:55204)
0806 06:30:49.727009 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0806 06:30:49.727036 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0806 06:30:49.727095 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:30:49.727211 (+ 116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:30:49.727212 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:30:49.727384 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0806 06:30:49.727432 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:30:49.727434 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:30:49.728284 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:30:49.728285 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:30:49.729089 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:30:49.729092 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:30:49.729508 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:30:49.729509 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:30:49.729565 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:30:49.729997 (+ 432us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 06:30:49.730002 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0806 06:30:49.731466 (+ 1464us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 06:30:49.732888 (+ 1422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:30:49.732889 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:30:49.732889 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:30:49.733050 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:30:49.733217 (+ 167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:30:49.733217 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:30:49.733218 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:30:49.733275 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:30:49.733515 (+ 240us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 06:30:49.733516 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 06:30:49.733562 (+ 46us) client_negotiation.cc:769] Sending connection context
0806 06:30:49.733622 (+ 60us) client_negotiation.cc:240] Negotiation successful
0806 06:30:49.733712 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20}
I20260806 06:30:49.735389 21986 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 527b23341eb44fa3807ee2a8af0b16d7: Bootstrap starting.
I20260806 06:30:49.736160 21986 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 527b23341eb44fa3807ee2a8af0b16d7: Neither blocks nor log segments found. Creating new log.
I20260806 06:30:49.736471 21986 log.cc:824] T 00000000000000000000000000000000 P 527b23341eb44fa3807ee2a8af0b16d7: Log is configured to *not* fsync() on all Append() calls
I20260806 06:30:49.737286 21986 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 527b23341eb44fa3807ee2a8af0b16d7: No bootstrap required, opened a new log
I20260806 06:30:49.738651 21986 raft_consensus.cc:348] T 00000000000000000000000000000000 P 527b23341eb44fa3807ee2a8af0b16d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "527b23341eb44fa3807ee2a8af0b16d7" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 33633 } } peers { permanent_uuid: "8b04b33b01dc4d07a4aa7b5a7470013b" member_type: VOTER last_known_addr { host: "127.21.12.189" port: 43499 } } peers { permanent_uuid: "b6869d74fc9d4af4b76a0146a1c8e1db" member_type: VOTER last_known_addr { host: "127.21.12.188" port: 45467 } }
I20260806 06:30:49.738793 21986 raft_consensus.cc:729] T 00000000000000000000000000000000 P 527b23341eb44fa3807ee2a8af0b16d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 527b23341eb44fa3807ee2a8af0b16d7, State: Initialized, Role: FOLLOWER
I20260806 06:30:49.738878 21986 consensus_queue.cc:260] T 00000000000000000000000000000000 P 527b23341eb44fa3807ee2a8af0b16d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "527b23341eb44fa3807ee2a8af0b16d7" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 33633 } } peers { permanent_uuid: "8b04b33b01dc4d07a4aa7b5a7470013b" member_type: VOTER last_known_addr { host: "127.21.12.189" port: 43499 } } peers { permanent_uuid: "b6869d74fc9d4af4b76a0146a1c8e1db" member_type: VOTER last_known_addr { host: "127.21.12.188" port: 45467 } }
I20260806 06:30:49.739154 22204 sys_catalog.cc:455] T 00000000000000000000000000000000 P 527b23341eb44fa3807ee2a8af0b16d7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "527b23341eb44fa3807ee2a8af0b16d7" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 33633 } } peers { permanent_uuid: "8b04b33b01dc4d07a4aa7b5a7470013b" member_type: VOTER last_known_addr { host: "127.21.12.189" port: 43499 } } peers { permanent_uuid: "b6869d74fc9d4af4b76a0146a1c8e1db" member_type: VOTER last_known_addr { host: "127.21.12.188" port: 45467 } } }
I20260806 06:30:49.739212 21986 sys_catalog.cc:564] T 00000000000000000000000000000000 P 527b23341eb44fa3807ee2a8af0b16d7 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 06:30:49.739236 21953 raft_consensus.cc:482] T 00000000000000000000000000000000 P 527b23341eb44fa3807ee2a8af0b16d7 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 06:30:49.739246 22204 sys_catalog.cc:458] T 00000000000000000000000000000000 P 527b23341eb44fa3807ee2a8af0b16d7 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 06:30:49.739303 21953 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 527b23341eb44fa3807ee2a8af0b16d7 [term 0 FOLLOWER]: Advancing to term 1
I20260806 06:30:49.739967 21953 raft_consensus.cc:504] T 00000000000000000000000000000000 P 527b23341eb44fa3807ee2a8af0b16d7 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "527b23341eb44fa3807ee2a8af0b16d7" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 33633 } } peers { permanent_uuid: "8b04b33b01dc4d07a4aa7b5a7470013b" member_type: VOTER last_known_addr { host: "127.21.12.189" port: 43499 } } peers { permanent_uuid: "b6869d74fc9d4af4b76a0146a1c8e1db" member_type: VOTER last_known_addr { host: "127.21.12.188" port: 45467 } }
I20260806 06:30:49.740164 21953 leader_election.cc:288] T 00000000000000000000000000000000 P 527b23341eb44fa3807ee2a8af0b16d7 [CANDIDATE]: Term 1 election: Requested vote from peers 8b04b33b01dc4d07a4aa7b5a7470013b (127.21.12.189:43499), b6869d74fc9d4af4b76a0146a1c8e1db (127.21.12.188:45467)
W20260806 06:30:49.741236 22222 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 527b23341eb44fa3807ee2a8af0b16d7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 06:30:49.741359 22222 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 06:30:49.741690 22143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "527b23341eb44fa3807ee2a8af0b16d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "b6869d74fc9d4af4b76a0146a1c8e1db"
I20260806 06:30:49.741838 22143 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b6869d74fc9d4af4b76a0146a1c8e1db [term 0 FOLLOWER]: Advancing to term 1
I20260806 06:30:49.742751 22143 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b6869d74fc9d4af4b76a0146a1c8e1db [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 527b23341eb44fa3807ee2a8af0b16d7 in term 1.
I20260806 06:30:49.743070 21916 leader_election.cc:302] T 00000000000000000000000000000000 P 527b23341eb44fa3807ee2a8af0b16d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 527b23341eb44fa3807ee2a8af0b16d7, b6869d74fc9d4af4b76a0146a1c8e1db; no voters:
I20260806 06:30:49.743355 22204 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 527b23341eb44fa3807ee2a8af0b16d7 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 06:30:49.743535 22204 raft_consensus.cc:686] T 00000000000000000000000000000000 P 527b23341eb44fa3807ee2a8af0b16d7 [term 1 LEADER]: Becoming Leader. State: Replica: 527b23341eb44fa3807ee2a8af0b16d7, State: Running, Role: LEADER
I20260806 06:30:49.743636 22204 consensus_queue.cc:237] T 00000000000000000000000000000000 P 527b23341eb44fa3807ee2a8af0b16d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "527b23341eb44fa3807ee2a8af0b16d7" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 33633 } } peers { permanent_uuid: "8b04b33b01dc4d07a4aa7b5a7470013b" member_type: VOTER last_known_addr { host: "127.21.12.189" port: 43499 } } peers { permanent_uuid: "b6869d74fc9d4af4b76a0146a1c8e1db" member_type: VOTER last_known_addr { host: "127.21.12.188" port: 45467 } }
I20260806 06:30:49.744035 22204 sys_catalog.cc:455] T 00000000000000000000000000000000 P 527b23341eb44fa3807ee2a8af0b16d7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 527b23341eb44fa3807ee2a8af0b16d7. Latest consensus state: current_term: 1 leader_uuid: "527b23341eb44fa3807ee2a8af0b16d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "527b23341eb44fa3807ee2a8af0b16d7" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 33633 } } peers { permanent_uuid: "8b04b33b01dc4d07a4aa7b5a7470013b" member_type: VOTER last_known_addr { host: "127.21.12.189" port: 43499 } } peers { permanent_uuid: "b6869d74fc9d4af4b76a0146a1c8e1db" member_type: VOTER last_known_addr { host: "127.21.12.188" port: 45467 } } }
I20260806 06:30:49.744110 22204 sys_catalog.cc:458] T 00000000000000000000000000000000 P 527b23341eb44fa3807ee2a8af0b16d7 [sys.catalog]: This master's current role is: LEADER
I20260806 06:30:49.744387 21894 master_runner.cc:186] Error getting master registration for 127.21.12.189:43499: OK, instance_id { permanent_uuid: "8b04b33b01dc4d07a4aa7b5a7470013b" instance_seqno: 1785997849529280 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260806 06:30:49.746682 21894 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260806 06:30:49.750397 22236 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 06:30:49.750934 22236 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 06:30:49.752396 22143 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b6869d74fc9d4af4b76a0146a1c8e1db [term 1 FOLLOWER]: Refusing update from remote peer 527b23341eb44fa3807ee2a8af0b16d7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 06:30:49.752693 22204 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 527b23341eb44fa3807ee2a8af0b16d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6869d74fc9d4af4b76a0146a1c8e1db" member_type: VOTER last_known_addr { host: "127.21.12.188" port: 45467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 06:30:49.753950 22184 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6869d74fc9d4af4b76a0146a1c8e1db [sys.catalog]: SysCatalogTable state changed. Reason: New leader 527b23341eb44fa3807ee2a8af0b16d7. Latest consensus state: current_term: 1 leader_uuid: "527b23341eb44fa3807ee2a8af0b16d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "527b23341eb44fa3807ee2a8af0b16d7" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 33633 } } peers { permanent_uuid: "8b04b33b01dc4d07a4aa7b5a7470013b" member_type: VOTER last_known_addr { host: "127.21.12.189" port: 43499 } } peers { permanent_uuid: "b6869d74fc9d4af4b76a0146a1c8e1db" member_type: VOTER last_known_addr { host: "127.21.12.188" port: 45467 } } }
I20260806 06:30:49.754037 22184 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6869d74fc9d4af4b76a0146a1c8e1db [sys.catalog]: This master's current role is: FOLLOWER
I20260806 06:30:49.756244 22231 sys_catalog.cc:455] T 00000000000000000000000000000000 P 527b23341eb44fa3807ee2a8af0b16d7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "527b23341eb44fa3807ee2a8af0b16d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "527b23341eb44fa3807ee2a8af0b16d7" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 33633 } } peers { permanent_uuid: "8b04b33b01dc4d07a4aa7b5a7470013b" member_type: VOTER last_known_addr { host: "127.21.12.189" port: 43499 } } peers { permanent_uuid: "b6869d74fc9d4af4b76a0146a1c8e1db" member_type: VOTER last_known_addr { host: "127.21.12.188" port: 45467 } } }
I20260806 06:30:49.756368 22231 sys_catalog.cc:458] T 00000000000000000000000000000000 P 527b23341eb44fa3807ee2a8af0b16d7 [sys.catalog]: This master's current role is: LEADER
I20260806 06:30:49.756732 22184 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6869d74fc9d4af4b76a0146a1c8e1db [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "527b23341eb44fa3807ee2a8af0b16d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "527b23341eb44fa3807ee2a8af0b16d7" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 33633 } } peers { permanent_uuid: "8b04b33b01dc4d07a4aa7b5a7470013b" member_type: VOTER last_known_addr { host: "127.21.12.189" port: 43499 } } peers { permanent_uuid: "b6869d74fc9d4af4b76a0146a1c8e1db" member_type: VOTER last_known_addr { host: "127.21.12.188" port: 45467 } } }
I20260806 06:30:49.756819 22184 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6869d74fc9d4af4b76a0146a1c8e1db [sys.catalog]: This master's current role is: FOLLOWER
I20260806 06:30:49.757165 22238 mvcc.cc:205] Tried to move back new op lower bound from 7315447192583729152 to 7315447192550658048. Current Snapshot: MvccSnapshot[applied={T|T < 7315447192583729152}]
I20260806 06:30:49.757740 22240 mvcc.cc:205] Tried to move back new op lower bound from 7315447192583729152 to 7315447192550658048. Current Snapshot: MvccSnapshot[applied={T|T < 7315447192583729152}]
I20260806 06:30:49.758453 22236 catalog_manager.cc:1347] Generated new cluster ID: da4f23b0c1aa443e8ae7d7254af680f0
I20260806 06:30:49.758504 22236 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 06:30:49.761056 22082 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.12.188" port: 45467 } attempt: 3
I20260806 06:30:49.762835 22088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:49.742999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44080 (local address 127.21.12.189:43499)
0806 06:30:49.743212 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0806 06:30:49.743214 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:30:49.743222 (+ 8us) server_negotiation.cc:407] Connection header received
0806 06:30:49.743246 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:30:49.743247 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:30:49.743257 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:30:49.743310 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:30:49.747431 (+ 4121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:49.748156 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:49.749197 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:49.749504 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:49.749543 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:30:49.752340 (+ 2797us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:30:49.752342 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:30:49.752342 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 06:30:49.752349 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:30:49.754117 (+ 1768us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:30:49.759579 (+ 5462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:30:49.759581 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:30:49.759581 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:30:49.759650 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:30:49.760617 (+ 967us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:30:49.760618 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:30:49.760618 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:30:49.760774 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:30:49.760870 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0806 06:30:49.762639 (+ 1769us) server_negotiation.cc:299] Negotiation successful
0806 06:30:49.762732 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":188}
I20260806 06:30:49.763947 22182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:49.742147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55208 (local address 127.21.12.188:45467)
0806 06:30:49.742185 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0806 06:30:49.742187 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:30:49.743289 (+ 1102us) server_negotiation.cc:407] Connection header received
0806 06:30:49.743424 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:30:49.743425 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:30:49.743437 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:30:49.743465 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:30:49.745867 (+ 2402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:49.746622 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:49.750368 (+ 3746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:49.750665 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:49.750710 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:30:49.754700 (+ 3990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:30:49.754702 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:30:49.754702 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 06:30:49.754709 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:30:49.756479 (+ 1770us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:30:49.760310 (+ 3831us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:30:49.760323 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:30:49.760324 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:30:49.760386 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:30:49.762167 (+ 1781us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:30:49.762168 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:30:49.762168 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:30:49.762320 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:30:49.762406 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0806 06:30:49.763773 (+ 1367us) server_negotiation.cc:299] Negotiation successful
0806 06:30:49.763860 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
Aug 06 06:30:49 dist-test-slave-4tdj krb5kdc[21876](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997849, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.189@KRBTEST.COM for kudu/127.21.12.188@KRBTEST.COM
I20260806 06:30:49.772298 22085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:49.761289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.188:45467 (local address 127.0.0.1:55222)
0806 06:30:49.761366 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0806 06:30:49.761375 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0806 06:30:49.761407 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:30:49.764190 (+ 2783us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:30:49.764192 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:30:49.764347 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0806 06:30:49.764390 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:30:49.764392 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:30:49.765356 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:30:49.765357 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:30:49.766187 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:30:49.766190 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:30:49.766758 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:30:49.766759 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:30:49.766829 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:30:49.767328 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 06:30:49.767333 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0806 06:30:49.768798 (+ 1465us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 06:30:49.771113 (+ 2315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:30:49.771114 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:30:49.771115 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:30:49.771324 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:30:49.771591 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:30:49.771591 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:30:49.771592 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:30:49.771649 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:30:49.771986 (+ 337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 06:30:49.771986 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 06:30:49.772051 (+ 65us) client_negotiation.cc:769] Sending connection context
0806 06:30:49.772104 (+ 53us) client_negotiation.cc:240] Negotiation successful
0806 06:30:49.772199 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61}
I20260806 06:30:49.772811 22182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:49.761467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55222 (local address 127.21.12.188:45467)
0806 06:30:49.764029 (+ 2562us) server_negotiation.cc:206] Beginning negotiation
0806 06:30:49.764031 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:30:49.764039 (+ 8us) server_negotiation.cc:407] Connection header received
0806 06:30:49.764058 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:30:49.764059 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:30:49.764070 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:30:49.764115 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:30:49.764479 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:49.765254 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:49.766284 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:49.766639 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:49.766697 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:30:49.768898 (+ 2201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:30:49.768900 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:30:49.768900 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 06:30:49.768907 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:30:49.770992 (+ 2085us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:30:49.771411 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:30:49.771412 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:30:49.771413 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:30:49.771492 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:30:49.771717 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:30:49.771718 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:30:49.771718 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:30:49.771909 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:30:49.772479 (+ 570us) server_negotiation.cc:1035] Waiting for connection context
0806 06:30:49.772549 (+ 70us) server_negotiation.cc:299] Negotiation successful
0806 06:30:49.772670 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2546,"thread_start_us":79,"threads_started":1}
I20260806 06:30:49.774459 22082 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b04b33b01dc4d07a4aa7b5a7470013b: Bootstrap starting.
I20260806 06:30:49.775219 22082 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8b04b33b01dc4d07a4aa7b5a7470013b: Neither blocks nor log segments found. Creating new log.
I20260806 06:30:49.775532 22082 log.cc:824] T 00000000000000000000000000000000 P 8b04b33b01dc4d07a4aa7b5a7470013b: Log is configured to *not* fsync() on all Append() calls
I20260806 06:30:49.776983 21989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:49.742609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49720 (local address 127.21.12.190:33633)
0806 06:30:49.742644 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0806 06:30:49.742646 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:30:49.743768 (+ 1122us) server_negotiation.cc:407] Connection header received
0806 06:30:49.743973 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:30:49.743975 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:30:49.743996 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:30:49.744023 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:30:49.746871 (+ 2848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:49.747571 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:49.750819 (+ 3248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:49.751087 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:49.751131 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:30:49.756952 (+ 5821us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:30:49.756954 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:30:49.756954 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 06:30:49.756961 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:30:49.758822 (+ 1861us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:30:49.761839 (+ 3017us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:30:49.761841 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:30:49.761841 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:30:49.761900 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:30:49.763359 (+ 1459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:30:49.763361 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:30:49.763361 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:30:49.763511 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:30:49.763606 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0806 06:30:49.776800 (+ 13194us) server_negotiation.cc:299] Negotiation successful
0806 06:30:49.776889 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260806 06:30:49.779037 22082 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b04b33b01dc4d07a4aa7b5a7470013b: No bootstrap required, opened a new log
I20260806 06:30:49.781072 22082 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8b04b33b01dc4d07a4aa7b5a7470013b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "527b23341eb44fa3807ee2a8af0b16d7" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 33633 } } peers { permanent_uuid: "8b04b33b01dc4d07a4aa7b5a7470013b" member_type: VOTER last_known_addr { host: "127.21.12.189" port: 43499 } } peers { permanent_uuid: "b6869d74fc9d4af4b76a0146a1c8e1db" member_type: VOTER last_known_addr { host: "127.21.12.188" port: 45467 } }
I20260806 06:30:49.781214 22082 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8b04b33b01dc4d07a4aa7b5a7470013b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b04b33b01dc4d07a4aa7b5a7470013b, State: Initialized, Role: FOLLOWER
I20260806 06:30:49.781307 22082 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8b04b33b01dc4d07a4aa7b5a7470013b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "527b23341eb44fa3807ee2a8af0b16d7" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 33633 } } peers { permanent_uuid: "8b04b33b01dc4d07a4aa7b5a7470013b" member_type: VOTER last_known_addr { host: "127.21.12.189" port: 43499 } } peers { permanent_uuid: "b6869d74fc9d4af4b76a0146a1c8e1db" member_type: VOTER last_known_addr { host: "127.21.12.188" port: 45467 } }
I20260806 06:30:49.781635 22082 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8b04b33b01dc4d07a4aa7b5a7470013b [sys.catalog]: configured and running, proceeding with master startup.
I20260806 06:30:49.781777 22045 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8b04b33b01dc4d07a4aa7b5a7470013b [term 0 FOLLOWER]: Advancing to term 1
I20260806 06:30:49.782346 22046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "527b23341eb44fa3807ee2a8af0b16d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "8b04b33b01dc4d07a4aa7b5a7470013b"
I20260806 06:30:49.782692 22045 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8b04b33b01dc4d07a4aa7b5a7470013b [term 1 FOLLOWER]: Refusing update from remote peer 527b23341eb44fa3807ee2a8af0b16d7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 06:30:49.783058 22204 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 527b23341eb44fa3807ee2a8af0b16d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b04b33b01dc4d07a4aa7b5a7470013b" member_type: VOTER last_known_addr { host: "127.21.12.189" port: 43499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 06:30:49.783975 22046 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8b04b33b01dc4d07a4aa7b5a7470013b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 527b23341eb44fa3807ee2a8af0b16d7 in term 1.
I20260806 06:30:49.786557 22254 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b04b33b01dc4d07a4aa7b5a7470013b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "527b23341eb44fa3807ee2a8af0b16d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "527b23341eb44fa3807ee2a8af0b16d7" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 33633 } } peers { permanent_uuid: "8b04b33b01dc4d07a4aa7b5a7470013b" member_type: VOTER last_known_addr { host: "127.21.12.189" port: 43499 } } peers { permanent_uuid: "b6869d74fc9d4af4b76a0146a1c8e1db" member_type: VOTER last_known_addr { host: "127.21.12.188" port: 45467 } } }
I20260806 06:30:49.786644 22254 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b04b33b01dc4d07a4aa7b5a7470013b [sys.catalog]: This master's current role is: FOLLOWER
I20260806 06:30:49.786688 22254 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b04b33b01dc4d07a4aa7b5a7470013b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 527b23341eb44fa3807ee2a8af0b16d7. Latest consensus state: current_term: 1 leader_uuid: "527b23341eb44fa3807ee2a8af0b16d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "527b23341eb44fa3807ee2a8af0b16d7" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 33633 } } peers { permanent_uuid: "8b04b33b01dc4d07a4aa7b5a7470013b" member_type: VOTER last_known_addr { host: "127.21.12.189" port: 43499 } } peers { permanent_uuid: "b6869d74fc9d4af4b76a0146a1c8e1db" member_type: VOTER last_known_addr { host: "127.21.12.188" port: 45467 } } }
I20260806 06:30:49.786722 22254 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b04b33b01dc4d07a4aa7b5a7470013b [sys.catalog]: This master's current role is: FOLLOWER
I20260806 06:30:49.786744 22254 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b04b33b01dc4d07a4aa7b5a7470013b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "527b23341eb44fa3807ee2a8af0b16d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "527b23341eb44fa3807ee2a8af0b16d7" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 33633 } } peers { permanent_uuid: "8b04b33b01dc4d07a4aa7b5a7470013b" member_type: VOTER last_known_addr { host: "127.21.12.189" port: 43499 } } peers { permanent_uuid: "b6869d74fc9d4af4b76a0146a1c8e1db" member_type: VOTER last_known_addr { host: "127.21.12.188" port: 45467 } } }
I20260806 06:30:49.786773 22254 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b04b33b01dc4d07a4aa7b5a7470013b [sys.catalog]: This master's current role is: FOLLOWER
I20260806 06:30:49.787187 22255 mvcc.cc:205] Tried to move back new op lower bound from 7315447192583729152 to 7315447192550658048. Current Snapshot: MvccSnapshot[applied={T|T < 7315447192583729152}]
I20260806 06:30:49.788146 22231 sys_catalog.cc:455] T 00000000000000000000000000000000 P 527b23341eb44fa3807ee2a8af0b16d7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "527b23341eb44fa3807ee2a8af0b16d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "527b23341eb44fa3807ee2a8af0b16d7" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 33633 } } peers { permanent_uuid: "8b04b33b01dc4d07a4aa7b5a7470013b" member_type: VOTER last_known_addr { host: "127.21.12.189" port: 43499 } } peers { permanent_uuid: "b6869d74fc9d4af4b76a0146a1c8e1db" member_type: VOTER last_known_addr { host: "127.21.12.188" port: 45467 } } }
I20260806 06:30:49.788237 22231 sys_catalog.cc:458] T 00000000000000000000000000000000 P 527b23341eb44fa3807ee2a8af0b16d7 [sys.catalog]: This master's current role is: LEADER
I20260806 06:30:49.789252 22236 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 06:30:49.789968 22236 catalog_manager.cc:1504] Loading token signing keys...
W20260806 06:30:49.793789 22267 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8b04b33b01dc4d07a4aa7b5a7470013b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 06:30:49.793850 22267 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 06:30:49.802338 22236 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 527b23341eb44fa3807ee2a8af0b16d7: Generated new TSK 0
I20260806 06:30:49.802577 22236 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260806 06:30:49.826032 22088 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 06:30:49.824420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44086 (local address 127.21.12.189:43499)
0806 06:30:49.824465 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0806 06:30:49.824468 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 06:30:49.825399 (+ 931us) server_negotiation.cc:407] Connection header received
0806 06:30:49.825505 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:30:49.825506 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:30:49.825517 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:30:49.825546 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:30:49.825965 (+ 419us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44086: BlockingRecv error: recv got EOF from 127.0.0.1:44086 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260806 06:30:49.827080 22245 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 06:30:49.824780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55238 (local address 127.21.12.188:45467)
0806 06:30:49.824827 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0806 06:30:49.824829 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:30:49.826510 (+ 1681us) server_negotiation.cc:407] Connection header received
0806 06:30:49.826612 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:30:49.826613 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:30:49.826625 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:30:49.826669 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:30:49.826987 (+ 318us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55238: BlockingRecv error: recv got EOF from 127.0.0.1:55238 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260806 06:30:49.828053 21989 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 06:30:49.825112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49722 (local address 127.21.12.190:33633)
0806 06:30:49.825152 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0806 06:30:49.825153 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0806 06:30:49.827547 (+ 2394us) server_negotiation.cc:407] Connection header received
0806 06:30:49.827643 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:30:49.827644 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:30:49.827655 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:30:49.827683 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:30:49.827981 (+ 298us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49722: BlockingRecv error: recv got EOF from 127.0.0.1:49722 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260806 06:30:49.830418 22088 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 06:30:49.829657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44100 (local address 127.21.12.189:43499)
0806 06:30:49.829692 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0806 06:30:49.829693 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0806 06:30:49.829884 (+ 191us) server_negotiation.cc:407] Connection header received
0806 06:30:49.829981 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:30:49.829982 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:30:49.830000 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:30:49.830027 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:30:49.830372 (+ 345us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44100: BlockingRecv error: recv got EOF from 127.0.0.1:44100 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260806 06:30:49.833448 22088 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 06:30:49.832706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44104 (local address 127.21.12.189:43499)
0806 06:30:49.832747 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0806 06:30:49.832749 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:30:49.832969 (+ 220us) server_negotiation.cc:407] Connection header received
0806 06:30:49.833074 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:30:49.833075 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:30:49.833086 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:30:49.833112 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:30:49.833408 (+ 296us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44104: BlockingRecv error: recv got EOF from 127.0.0.1:44104 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260806 06:30:49.837262 22088 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 06:30:49.836565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44116 (local address 127.21.12.189:43499)
0806 06:30:49.836596 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0806 06:30:49.836598 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:30:49.836805 (+ 207us) server_negotiation.cc:407] Connection header received
0806 06:30:49.836891 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:30:49.836892 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:30:49.836901 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:30:49.836928 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:30:49.837223 (+ 295us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44116: BlockingRecv error: recv got EOF from 127.0.0.1:44116 (error 108)
Metrics: {"server-negotiator.queue_time_us":14}
W20260806 06:30:49.843138 22088 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 06:30:49.842414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44130 (local address 127.21.12.189:43499)
0806 06:30:49.842446 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0806 06:30:49.842447 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0806 06:30:49.842659 (+ 212us) server_negotiation.cc:407] Connection header received
0806 06:30:49.842752 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:30:49.842753 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:30:49.842764 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:30:49.842797 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:30:49.843098 (+ 301us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44130: BlockingRecv error: recv got EOF from 127.0.0.1:44130 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260806 06:30:49.853121 22088 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 06:30:49.852315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44140 (local address 127.21.12.189:43499)
0806 06:30:49.852350 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0806 06:30:49.852352 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:30:49.852593 (+ 241us) server_negotiation.cc:407] Connection header received
0806 06:30:49.852687 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:30:49.852688 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:30:49.852699 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:30:49.852726 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:30:49.853064 (+ 338us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44140: BlockingRecv error: recv got EOF from 127.0.0.1:44140 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260806 06:30:49.871410 22088 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 06:30:49.870554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44152 (local address 127.21.12.189:43499)
0806 06:30:49.870597 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0806 06:30:49.870600 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 06:30:49.870848 (+ 248us) server_negotiation.cc:407] Connection header received
0806 06:30:49.870940 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:30:49.870941 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:30:49.870952 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:30:49.870979 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:30:49.871349 (+ 370us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44152: BlockingRecv error: recv got EOF from 127.0.0.1:44152 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260806 06:30:49.905457 22088 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 06:30:49.904773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44162 (local address 127.21.12.189:43499)
0806 06:30:49.904820 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0806 06:30:49.904823 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 06:30:49.904943 (+ 120us) server_negotiation.cc:407] Connection header received
0806 06:30:49.905013 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:30:49.905014 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:30:49.905026 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:30:49.905067 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:30:49.905396 (+ 329us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44162: BlockingRecv error: recv got EOF from 127.0.0.1:44162 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260806 06:30:49.971176 22088 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 06:30:49.970495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44176 (local address 127.21.12.189:43499)
0806 06:30:49.970568 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0806 06:30:49.970571 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 06:30:49.970679 (+ 108us) server_negotiation.cc:407] Connection header received
0806 06:30:49.970732 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:30:49.970733 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:30:49.970745 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:30:49.970796 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:30:49.971116 (+ 320us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44176: BlockingRecv error: recv got EOF from 127.0.0.1:44176 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260806 06:30:50.100946 22088 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 06:30:50.100331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44178 (local address 127.21.12.189:43499)
0806 06:30:50.100413 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0806 06:30:50.100415 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:30:50.100465 (+ 50us) server_negotiation.cc:407] Connection header received
0806 06:30:50.100511 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:30:50.100512 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:30:50.100526 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:30:50.100589 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:30:50.100898 (+ 309us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44178: BlockingRecv error: recv got EOF from 127.0.0.1:44178 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260806 06:30:50.358741 22088 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 06:30:50.358095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44182 (local address 127.21.12.189:43499)
0806 06:30:50.358164 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0806 06:30:50.358166 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:30:50.358190 (+ 24us) server_negotiation.cc:407] Connection header received
0806 06:30:50.358266 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:30:50.358267 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:30:50.358280 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:30:50.358356 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:30:50.358678 (+ 322us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44182: BlockingRecv error: recv got EOF from 127.0.0.1:44182 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
I20260806 06:30:50.717303 22199 catalog_manager.cc:1259] Loaded cluster ID: da4f23b0c1aa443e8ae7d7254af680f0
I20260806 06:30:50.717381 22199 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b6869d74fc9d4af4b76a0146a1c8e1db: loading cluster ID for follower catalog manager: success
I20260806 06:30:50.718147 22199 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b6869d74fc9d4af4b76a0146a1c8e1db: acquiring CA information for follower catalog manager: success
I20260806 06:30:50.718463 22199 catalog_manager.cc:1602] T 00000000000000000000000000000000 P b6869d74fc9d4af4b76a0146a1c8e1db: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260806 06:30:50.794277 22267 catalog_manager.cc:1259] Loaded cluster ID: da4f23b0c1aa443e8ae7d7254af680f0
I20260806 06:30:50.794373 22267 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 8b04b33b01dc4d07a4aa7b5a7470013b: loading cluster ID for follower catalog manager: success
I20260806 06:30:50.795188 22267 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 8b04b33b01dc4d07a4aa7b5a7470013b: acquiring CA information for follower catalog manager: success
I20260806 06:30:50.795485 22267 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 8b04b33b01dc4d07a4aa7b5a7470013b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260806 06:30:50.874326 22352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:50.871743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44190 (local address 127.21.12.189:43499)
0806 06:30:50.871869 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0806 06:30:50.871870 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0806 06:30:50.871881 (+ 11us) server_negotiation.cc:407] Connection header received
0806 06:30:50.871940 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:30:50.871941 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:30:50.872008 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:30:50.872073 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 06:30:50.872223 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:50.872724 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:50.873473 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:50.873683 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:50.873737 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:30:50.873938 (+ 201us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 06:30:50.874062 (+ 124us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 06:30:50.874132 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0806 06:30:50.874189 (+ 57us) server_negotiation.cc:299] Negotiation successful
0806 06:30:50.874211 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":61,"threads_started":1}
I20260806 06:30:50.877867 22361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:50.875918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55244 (local address 127.21.12.188:45467)
0806 06:30:50.876025 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0806 06:30:50.876026 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0806 06:30:50.876051 (+ 25us) server_negotiation.cc:407] Connection header received
0806 06:30:50.876085 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:30:50.876086 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:30:50.876124 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:30:50.876180 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 06:30:50.876246 (+ 66us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:50.876712 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:50.877341 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:50.877512 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:50.877551 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:30:50.877646 (+ 95us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 06:30:50.877737 (+ 91us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 06:30:50.877782 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0806 06:30:50.877796 (+ 14us) server_negotiation.cc:299] Negotiation successful
0806 06:30:50.877811 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85,"thread_start_us":53,"threads_started":1}
I20260806 06:30:50.881732 22361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:50.879426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55252 (local address 127.21.12.188:45467)
0806 06:30:50.879457 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0806 06:30:50.879459 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:30:50.879465 (+ 6us) server_negotiation.cc:407] Connection header received
0806 06:30:50.879524 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:30:50.879524 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:30:50.879533 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:30:50.879578 (+ 45us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 06:30:50.879703 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:50.880393 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:50.881102 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:50.881292 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:50.881343 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:30:50.881512 (+ 169us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 06:30:50.881541 (+ 29us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 06:30:50.881590 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0806 06:30:50.881644 (+ 54us) server_negotiation.cc:299] Negotiation successful
0806 06:30:50.881671 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260806 06:30:50.882089 22352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:50.879241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44206 (local address 127.21.12.189:43499)
0806 06:30:50.879278 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0806 06:30:50.879280 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:30:50.879371 (+ 91us) server_negotiation.cc:407] Connection header received
0806 06:30:50.879468 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:30:50.879469 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:30:50.879480 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:30:50.879505 (+ 25us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 06:30:50.879652 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:50.880369 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:50.881362 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:50.881629 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:50.881666 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:30:50.881872 (+ 206us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 06:30:50.881901 (+ 29us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 06:30:50.881950 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0806 06:30:50.882002 (+ 52us) server_negotiation.cc:299] Negotiation successful
0806 06:30:50.882023 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260806 06:30:50.882521 22370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:50.879271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49738 (local address 127.21.12.190:33633)
0806 06:30:50.879369 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0806 06:30:50.879370 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0806 06:30:50.879472 (+ 102us) server_negotiation.cc:407] Connection header received
0806 06:30:50.879604 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:30:50.879605 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:30:50.879659 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:30:50.879729 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 06:30:50.879814 (+ 85us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:50.880291 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:50.881347 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:50.881636 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:50.881725 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:30:50.882130 (+ 405us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 06:30:50.882238 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 06:30:50.882321 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0806 06:30:50.882405 (+ 84us) server_negotiation.cc:299] Negotiation successful
0806 06:30:50.882438 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77,"thread_start_us":45,"threads_started":1}
I20260806 06:30:50.884002 21554 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHhLhT/build/release/bin/kudu with pid 21894
I20260806 06:30:50.895292 21554 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHhLhT/build/release/bin/kudu with pid 21993
I20260806 06:30:50.903144 21554 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHhLhT/build/release/bin/kudu with pid 22091
2026-08-06T06:30:50Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (2874 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785997581049705-21554-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 06:30:50 dist-test-slave-4tdj krb5kdc[22378](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 06:30:50 dist-test-slave-4tdj krb5kdc[22378](info): set up 2 sockets
Aug 06 06:30:50 dist-test-slave-4tdj krb5kdc[22378](info): commencing operation
krb5kdc: starting...
W20260806 06:30:52.028513 21554 mini_kdc.cc:121] Time spent starting KDC: real 1.100s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785997581049705-21554-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785997581049705-21554-0/krb5kdc/test-user.keytab.
Aug 06 06:30:52 dist-test-slave-4tdj krb5kdc[22378](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997852, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T06:30:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T06:30:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.21.12.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.12.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.12.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.190.keytab.
Entry for principal kudu/127.21.12.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.190.keytab.
I20260806 06:30:52.144217 21554 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHhLhT/build/release/bin/kudu
/tmp/dist-test-taskFHhLhT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785997581049705-21554-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785997581049705-21554-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785997581049705-21554-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785997581049705-21554-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.12.190:44049
--webserver_interface=127.21.12.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.190.keytab
--principal=kudu/127.21.12.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.148:33149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.12.190:44049,127.21.12.189:40089,127.21.12.188:41667
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785997581049705-21554-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785997581049705-21554-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785997581049705-21554-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 06:30:52.228626 22394 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260806 06:30:52.228912 22394 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 06:30:52.228943 22394 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 06:30:52.229028 22394 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 06:30:52.230722 22394 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 06:30:52.230779 22394 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 06:30:52.230799 22394 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 06:30:52.230818 22394 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 06:30:52.230839 22394 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 06:30:52.233014 22394 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.148:33149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785997581049705-21554-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785997581049705-21554-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.12.190:44049,127.21.12.189:40089,127.21.12.188:41667
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.190.keytab
--principal=kudu/127.21.12.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.12.190:44049
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785997581049705-21554-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.12.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22394
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785997581049705-21554-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 05:11:02 UTC on ccycloud.quasar-hvzbbz.root.comops.site
build id 4569617
I20260806 06:30:52.233285 22394 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 06:30:52.233547 22394 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 06:30:52.240537 22401 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:30:52.240749 22394 server_base.cc:1034] running on GCE node
W20260806 06:30:52.240578 22405 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:30:52.240926 22402 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:30:52.241205 22394 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 06:30:52.241506 22394 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 06:30:52.242645 22394 hybrid_clock.cc:648] HybridClock initialized: now 1785997852242629 us; error 37 us; skew 500 ppm
Aug 06 06:30:52 dist-test-slave-4tdj krb5kdc[22378](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997852, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 06:30:52.245698 22394 init.cc:377] Logged in from keytab as kudu/127.21.12.190@KRBTEST.COM (short username kudu)
I20260806 06:30:52.246254 22394 webserver.cc:460] Webserver started at http://127.21.12.190:43969/ using document root <none> and password file <none>
I20260806 06:30:52.246582 22394 fs_manager.cc:359] Metadata directory not provided
I20260806 06:30:52.246650 22394 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 06:30:52.246794 22394 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 06:30:52.248179 22394 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785997581049705-21554-0/minicluster-data/master-0/data/instance:
uuid: "3179d6681ac24153ab6385dbba22d241"
format_stamp: "Formatted at 2026-08-06 06:30:52 on dist-test-slave-4tdj"
server_key: "e669255649e769ebc39da9b58e70e861"
server_key_iv: "4b346dba24e8d68a5d7822af613f25f4"
server_key_version: "encryptionkey@0"
I20260806 06:30:52.248634 22394 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785997581049705-21554-0/minicluster-data/master-0/wal/instance:
uuid: "3179d6681ac24153ab6385dbba22d241"
format_stamp: "Formatted at 2026-08-06 06:30:52 on dist-test-slave-4tdj"
server_key: "e669255649e769ebc39da9b58e70e861"
server_key_iv: "4b346dba24e8d68a5d7822af613f25f4"
server_key_version: "encryptionkey@0"
I20260806 06:30:52.250375 22394 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.004s
I20260806 06:30:52.251325 22413 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 06:30:52.251513 22394 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 06:30:52.251591 22394 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785997581049705-21554-0/minicluster-data/master-0/data,/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785997581049705-21554-0/minicluster-data/master-0/wal
uuid: "3179d6681ac24153ab6385dbba22d241"
format_stamp: "Formatted at 2026-08-06 06:30:52 on dist-test-slave-4tdj"
server_key: "e669255649e769ebc39da9b58e70e861"
server_key_iv: "4b346dba24e8d68a5d7822af613f25f4"
server_key_version: "encryptionkey@0"
I20260806 06:30:52.251673 22394 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785997581049705-21554-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785997581049705-21554-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785997581049705-21554-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 06:30:52.265916 22394 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 06:30:52.267699 22394 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 06:30:52.267809 22394 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 06:30:52.271569 22394 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.190:44049
I20260806 06:30:52.271613 22479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.190:44049 every 8 connection(s)
I20260806 06:30:52.271956 22394 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785997581049705-21554-0/minicluster-data/master-0/data/info.pb
I20260806 06:30:52.272949 22480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 06:30:52.274552 22480 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.12.190" port: 44049 } has no permanent_uuid. Determining permanent_uuid...
I20260806 06:30:52.278972 21554 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHhLhT/build/release/bin/kudu as pid 22394
I20260806 06:30:52.279072 21554 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785997581049705-21554-0/minicluster-data/master-0/wal/instance
I20260806 06:30:52.279271 21554 external_mini_cluster.cc:1428] Setting key cc430f7c63cd43c1e9b7839fa45ac24b
Aug 06 06:30:52 dist-test-slave-4tdj krb5kdc[22378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997852, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.190@KRBTEST.COM for kudu/127.21.12.190@KRBTEST.COM
I20260806 06:30:52.283627 22483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:52.275216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:44049 (local address 127.0.0.1:34980)
0806 06:30:52.275411 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0806 06:30:52.275449 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0806 06:30:52.275895 (+ 446us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:30:52.276120 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:30:52.276123 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:30:52.276360 (+ 237us) client_negotiation.cc:189] Negotiated authn=SASL
0806 06:30:52.276511 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:30:52.276515 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:30:52.277448 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:30:52.277449 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:30:52.278117 (+ 668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:30:52.278119 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:30:52.278637 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:30:52.278637 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:30:52.278693 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:30:52.279203 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 06:30:52.279211 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0806 06:30:52.280690 (+ 1479us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 06:30:52.282189 (+ 1499us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:30:52.282195 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:30:52.282197 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:30:52.282428 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:30:52.282631 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:30:52.282632 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:30:52.282632 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:30:52.282721 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:30:52.283081 (+ 360us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 06:30:52.283087 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 06:30:52.283150 (+ 63us) client_negotiation.cc:769] Sending connection context
0806 06:30:52.283230 (+ 80us) client_negotiation.cc:240] Negotiation successful
0806 06:30:52.283397 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":107,"threads_started":1}
I20260806 06:30:52.283627 22484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:52.275239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34980 (local address 127.21.12.190:44049)
0806 06:30:52.275438 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0806 06:30:52.275448 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 06:30:52.275533 (+ 85us) server_negotiation.cc:407] Connection header received
0806 06:30:52.276008 (+ 475us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:30:52.276010 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:30:52.276026 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:30:52.276104 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:30:52.276578 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:52.277378 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:52.278189 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:52.278524 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:52.278602 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:30:52.280770 (+ 2168us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:30:52.280777 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:30:52.280777 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 06:30:52.280782 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:30:52.282113 (+ 1331us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:30:52.282502 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:30:52.282505 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:30:52.282505 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:30:52.282576 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:30:52.282775 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:30:52.282776 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:30:52.282776 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:30:52.283015 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:30:52.283088 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0806 06:30:52.283284 (+ 196us) server_negotiation.cc:299] Negotiation successful
0806 06:30:52.283454 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":88,"server-negotiator.queue_time_us":160,"thread_start_us":115,"threads_started":1}
I20260806 06:30:52.284531 22480 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.12.189" port: 40089 } has no permanent_uuid. Determining permanent_uuid...
I20260806 06:30:52.284857 22483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:52.284752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.189:40089 (local address 127.0.0.1:55954)
0806 06:30:52.284794 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0806 06:30:52.284837 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.189:40089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260806 06:30:52.284982 22418 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.12.189:40089: connect: Connection refused (error 111)
I20260806 06:30:52.285400 22483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:52.285313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.189:40089 (local address 127.0.0.1:55966)
0806 06:30:52.285353 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0806 06:30:52.285378 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.189:40089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260806 06:30:52.285494 22480 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.12.189:40089: Network error: Client connection negotiation failed: client connection to 127.21.12.189:40089: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.21.12.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.12.189@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.12.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.189.keytab.
Entry for principal kudu/127.21.12.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.189.keytab.
I20260806 06:30:52.303067 21554 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHhLhT/build/release/bin/kudu
/tmp/dist-test-taskFHhLhT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785997581049705-21554-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785997581049705-21554-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785997581049705-21554-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785997581049705-21554-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.12.189:40089
--webserver_interface=127.21.12.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.189.keytab
--principal=kudu/127.21.12.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.148:33149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.12.190:44049,127.21.12.189:40089,127.21.12.188:41667
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785997581049705-21554-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785997581049705-21554-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785997581049705-21554-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 06:30:52.334702 22480 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.12.189" port: 40089 } attempt: 1
I20260806 06:30:52.335215 22483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:52.335079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.189:40089 (local address 127.0.0.1:55974)
0806 06:30:52.335143 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0806 06:30:52.335186 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.189:40089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260806 06:30:52.335551 22483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:52.335471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.189:40089 (local address 127.0.0.1:55986)
0806 06:30:52.335509 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0806 06:30:52.335533 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.189:40089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260806 06:30:52.335629 22480 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.12.189:40089: Network error: Client connection negotiation failed: client connection to 127.21.12.189:40089: connect: Connection refused (error 111)
W20260806 06:30:52.384271 22489 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260806 06:30:52.384464 22489 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 06:30:52.384480 22489 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 06:30:52.384500 22489 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 06:30:52.385643 22489 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 06:30:52.385684 22489 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 06:30:52.385696 22489 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 06:30:52.385720 22489 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 06:30:52.385742 22489 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 06:30:52.387331 22489 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.148:33149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785997581049705-21554-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785997581049705-21554-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.21.12.190:44049,127.21.12.189:40089,127.21.12.188:41667
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.189.keytab
--principal=kudu/127.21.12.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.12.189:40089
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785997581049705-21554-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.21.12.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22489
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785997581049705-21554-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 05:11:02 UTC on ccycloud.quasar-hvzbbz.root.comops.site
build id 4569617
I20260806 06:30:52.387560 22489 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 06:30:52.387770 22489 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 06:30:52.392242 22497 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:30:52.392321 22501 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:30:52.392256 22498 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:30:52.392436 22489 server_base.cc:1034] running on GCE node
I20260806 06:30:52.392633 22489 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 06:30:52.392869 22489 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 06:30:52.394006 22489 hybrid_clock.cc:648] HybridClock initialized: now 1785997852393996 us; error 26 us; skew 500 ppm
Aug 06 06:30:52 dist-test-slave-4tdj krb5kdc[22378](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997852, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 06:30:52.396341 22489 init.cc:377] Logged in from keytab as kudu/127.21.12.189@KRBTEST.COM (short username kudu)
I20260806 06:30:52.396714 22489 webserver.cc:460] Webserver started at http://127.21.12.189:41571/ using document root <none> and password file <none>
I20260806 06:30:52.396904 22489 fs_manager.cc:359] Metadata directory not provided
I20260806 06:30:52.396943 22489 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 06:30:52.397032 22489 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 06:30:52.397975 22489 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785997581049705-21554-0/minicluster-data/master-1/data/instance:
uuid: "0350b17ccfb24e5c82c3a2931bbb5caf"
format_stamp: "Formatted at 2026-08-06 06:30:52 on dist-test-slave-4tdj"
server_key: "2326754a7b584191ebf4648f3944baf1"
server_key_iv: "8ffe20d9763de9d0d5daf6a83897cd1f"
server_key_version: "encryptionkey@0"
I20260806 06:30:52.398260 22489 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785997581049705-21554-0/minicluster-data/master-1/wal/instance:
uuid: "0350b17ccfb24e5c82c3a2931bbb5caf"
format_stamp: "Formatted at 2026-08-06 06:30:52 on dist-test-slave-4tdj"
server_key: "2326754a7b584191ebf4648f3944baf1"
server_key_iv: "8ffe20d9763de9d0d5daf6a83897cd1f"
server_key_version: "encryptionkey@0"
I20260806 06:30:52.399422 22489 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260806 06:30:52.400008 22508 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 06:30:52.400169 22489 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260806 06:30:52.400254 22489 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785997581049705-21554-0/minicluster-data/master-1/data,/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785997581049705-21554-0/minicluster-data/master-1/wal
uuid: "0350b17ccfb24e5c82c3a2931bbb5caf"
format_stamp: "Formatted at 2026-08-06 06:30:52 on dist-test-slave-4tdj"
server_key: "2326754a7b584191ebf4648f3944baf1"
server_key_iv: "8ffe20d9763de9d0d5daf6a83897cd1f"
server_key_version: "encryptionkey@0"
I20260806 06:30:52.400311 22489 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785997581049705-21554-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785997581049705-21554-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785997581049705-21554-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 06:30:52.403764 22480 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.12.189" port: 40089 } attempt: 2
I20260806 06:30:52.404086 22483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:52.403983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.189:40089 (local address 127.0.0.1:55990)
0806 06:30:52.404030 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0806 06:30:52.404060 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.189:40089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260806 06:30:52.404376 22483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:52.404299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.189:40089 (local address 127.0.0.1:55994)
0806 06:30:52.404331 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0806 06:30:52.404355 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.189:40089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
W20260806 06:30:52.404448 22480 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.12.189:40089: Network error: Client connection negotiation failed: client connection to 127.21.12.189:40089: connect: Connection refused (error 111)
I20260806 06:30:52.414351 22489 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 06:30:52.415939 22489 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 06:30:52.416064 22489 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 06:30:52.419725 22489 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.189:40089
I20260806 06:30:52.419759 22585 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.189:40089 every 8 connection(s)
I20260806 06:30:52.420059 22489 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785997581049705-21554-0/minicluster-data/master-1/data/info.pb
I20260806 06:30:52.420979 22586 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 06:30:52.422513 22586 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.12.190" port: 44049 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 06:30:52 dist-test-slave-4tdj krb5kdc[22378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997852, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.189@KRBTEST.COM for kudu/127.21.12.190@KRBTEST.COM
I20260806 06:30:52.427879 21554 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHhLhT/build/release/bin/kudu as pid 22489
I20260806 06:30:52.427985 21554 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785997581049705-21554-0/minicluster-data/master-1/wal/instance
I20260806 06:30:52.428176 21554 external_mini_cluster.cc:1428] Setting key 090c5f6051726bbbc1de4ea5136e90db
I20260806 06:30:52.431062 22484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:52.423135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34988 (local address 127.21.12.190:44049)
0806 06:30:52.423190 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0806 06:30:52.423192 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:30:52.423385 (+ 193us) server_negotiation.cc:407] Connection header received
0806 06:30:52.423847 (+ 462us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:30:52.423849 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:30:52.423861 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:30:52.423912 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:30:52.424491 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:52.424961 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:52.425896 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:52.426092 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:52.426146 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:30:52.428318 (+ 2172us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:30:52.428320 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:30:52.428320 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 06:30:52.428329 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:30:52.429709 (+ 1380us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:30:52.430117 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:30:52.430118 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:30:52.430119 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:30:52.430172 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:30:52.430473 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:30:52.430474 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:30:52.430474 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:30:52.430584 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:30:52.430639 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0806 06:30:52.430891 (+ 252us) server_negotiation.cc:299] Negotiation successful
0806 06:30:52.430973 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260806 06:30:52.431226 22589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:52.423129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:44049 (local address 127.0.0.1:34988)
0806 06:30:52.423278 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0806 06:30:52.423311 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0806 06:30:52.423753 (+ 442us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:30:52.423974 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:30:52.423977 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:30:52.424218 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0806 06:30:52.424407 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:30:52.424411 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:30:52.425040 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:30:52.425041 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:30:52.425817 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:30:52.425819 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:30:52.426186 (+ 367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:30:52.426187 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:30:52.426360 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:30:52.426825 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 06:30:52.426833 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0806 06:30:52.428191 (+ 1358us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 06:30:52.429800 (+ 1609us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:30:52.429809 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:30:52.429811 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:30:52.430057 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:30:52.430240 (+ 183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:30:52.430241 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:30:52.430241 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:30:52.430413 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:30:52.430639 (+ 226us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 06:30:52.430646 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 06:30:52.430777 (+ 131us) client_negotiation.cc:769] Sending connection context
0806 06:30:52.430863 (+ 86us) client_negotiation.cc:240] Negotiation successful
0806 06:30:52.431004 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":76,"threads_started":1}
I20260806 06:30:52.432926 22586 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.12.189" port: 40089 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.21.12.188@KRBTEST.COM; defaulting to no policy
Aug 06 06:30:52 dist-test-slave-4tdj krb5kdc[22378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997852, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.189@KRBTEST.COM for kudu/127.21.12.189@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.21.12.188@KRBTEST.COM" created.
I20260806 06:30:52.439921 22589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:52.433173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.189:40089 (local address 127.0.0.1:55996)
0806 06:30:52.433224 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0806 06:30:52.433236 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 06:30:52.433277 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:30:52.433488 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:30:52.433489 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:30:52.433633 (+ 144us) client_negotiation.cc:189] Negotiated authn=SASL
0806 06:30:52.433697 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:30:52.433699 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:30:52.434536 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:30:52.434537 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:30:52.435303 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:30:52.435307 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:30:52.435735 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:30:52.435736 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:30:52.435812 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:30:52.436141 (+ 329us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 06:30:52.436146 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0806 06:30:52.437332 (+ 1186us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 06:30:52.438901 (+ 1569us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:30:52.438902 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:30:52.438902 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:30:52.439113 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:30:52.439295 (+ 182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:30:52.439295 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:30:52.439296 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:30:52.439347 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:30:52.439604 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 06:30:52.439605 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 06:30:52.439643 (+ 38us) client_negotiation.cc:769] Sending connection context
0806 06:30:52.439713 (+ 70us) client_negotiation.cc:240] Negotiation successful
0806 06:30:52.439812 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260806 06:30:52.439982 22591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:52.433226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55996 (local address 127.21.12.189:40089)
0806 06:30:52.433356 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0806 06:30:52.433361 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 06:30:52.433378 (+ 17us) server_negotiation.cc:407] Connection header received
0806 06:30:52.433398 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:30:52.433399 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:30:52.433409 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:30:52.433467 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:30:52.433763 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:52.434462 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:52.435379 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:52.435664 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:52.435726 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:30:52.437404 (+ 1678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:30:52.437413 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:30:52.437413 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 06:30:52.437419 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:30:52.438827 (+ 1408us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:30:52.439188 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:30:52.439191 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:30:52.439192 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:30:52.439244 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:30:52.439399 (+ 155us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:30:52.439400 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:30:52.439400 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:30:52.439546 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:30:52.439618 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0806 06:30:52.439751 (+ 133us) server_negotiation.cc:299] Negotiation successful
0806 06:30:52.439863 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":50,"server-negotiator.queue_time_us":104,"thread_start_us":69,"threads_started":1}
I20260806 06:30:52.440516 22586 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.12.188" port: 41667 } has no permanent_uuid. Determining permanent_uuid...
I20260806 06:30:52.440845 22589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:52.440742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.188:41667 (local address 127.0.0.1:47550)
0806 06:30:52.440779 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0806 06:30:52.440822 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.188:41667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
W20260806 06:30:52.440982 22510 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.12.188:41667: connect: Connection refused (error 111)
I20260806 06:30:52.441393 22589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:52.441306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.188:41667 (local address 127.0.0.1:47552)
0806 06:30:52.441344 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0806 06:30:52.441372 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.188:41667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260806 06:30:52.441506 22586 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.12.188:41667: Network error: Client connection negotiation failed: client connection to 127.21.12.188:41667: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.21.12.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.188.keytab.
Entry for principal kudu/127.21.12.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.188.keytab.
I20260806 06:30:52.453138 21554 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHhLhT/build/release/bin/kudu
/tmp/dist-test-taskFHhLhT/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785997581049705-21554-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785997581049705-21554-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785997581049705-21554-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785997581049705-21554-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.12.188:41667
--webserver_interface=127.21.12.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.188.keytab
--principal=kudu/127.21.12.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.21.12.148:33149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.12.190:44049,127.21.12.189:40089,127.21.12.188:41667
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785997581049705-21554-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785997581049705-21554-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785997581049705-21554-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 06:30:52.490778 22586 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.12.188" port: 41667 } attempt: 1
I20260806 06:30:52.491333 22589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:52.491191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.188:41667 (local address 127.0.0.1:47568)
0806 06:30:52.491252 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0806 06:30:52.491298 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.188:41667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260806 06:30:52.491662 22589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:52.491583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.188:41667 (local address 127.0.0.1:47582)
0806 06:30:52.491620 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0806 06:30:52.491645 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.188:41667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260806 06:30:52.491751 22586 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.12.188:41667: Network error: Client connection negotiation failed: client connection to 127.21.12.188:41667: connect: Connection refused (error 111)
I20260806 06:30:52.495572 22480 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.12.189" port: 40089 } attempt: 3
Aug 06 06:30:52 dist-test-slave-4tdj krb5kdc[22378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997852, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.190@KRBTEST.COM for kudu/127.21.12.189@KRBTEST.COM
I20260806 06:30:52.502400 22483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:52.495882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.189:40089 (local address 127.0.0.1:56008)
0806 06:30:52.495937 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0806 06:30:52.495949 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 06:30:52.496021 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:30:52.496169 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:30:52.496170 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:30:52.496321 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0806 06:30:52.496390 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:30:52.496392 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:30:52.497336 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:30:52.497337 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:30:52.497864 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:30:52.497866 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:30:52.498352 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:30:52.498353 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:30:52.498408 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:30:52.498816 (+ 408us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 06:30:52.498821 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0806 06:30:52.499994 (+ 1173us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 06:30:52.501544 (+ 1550us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:30:52.501545 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:30:52.501545 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:30:52.501702 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:30:52.501872 (+ 170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:30:52.501872 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:30:52.501872 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:30:52.501919 (+ 47us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:30:52.502144 (+ 225us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 06:30:52.502145 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 06:30:52.502191 (+ 46us) client_negotiation.cc:769] Sending connection context
0806 06:30:52.502234 (+ 43us) client_negotiation.cc:240] Negotiation successful
0806 06:30:52.502332 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260806 06:30:52.502457 22591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:52.495928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56008 (local address 127.21.12.189:40089)
0806 06:30:52.495996 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0806 06:30:52.495998 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:30:52.496009 (+ 11us) server_negotiation.cc:407] Connection header received
0806 06:30:52.496071 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:30:52.496072 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:30:52.496093 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:30:52.496152 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:30:52.496454 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:52.497246 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:52.497935 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:52.498262 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:52.498357 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:30:52.500066 (+ 1709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:30:52.500068 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:30:52.500068 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 06:30:52.500076 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:30:52.501464 (+ 1388us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:30:52.501759 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:30:52.501760 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:30:52.501760 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:30:52.501819 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:30:52.501965 (+ 146us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:30:52.501965 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:30:52.501966 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:30:52.502093 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:30:52.502149 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0806 06:30:52.502269 (+ 120us) server_negotiation.cc:299] Negotiation successful
0806 06:30:52.502370 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260806 06:30:52.502856 22480 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.12.188" port: 41667 } has no permanent_uuid. Determining permanent_uuid...
I20260806 06:30:52.503196 22483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:52.503099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.188:41667 (local address 127.0.0.1:47584)
0806 06:30:52.503139 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0806 06:30:52.503176 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.188:41667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260806 06:30:52.503453 22483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:52.503383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.188:41667 (local address 127.0.0.1:47598)
0806 06:30:52.503418 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0806 06:30:52.503439 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.188:41667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260806 06:30:52.503538 22480 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.12.188:41667: Network error: Client connection negotiation failed: client connection to 127.21.12.188:41667: connect: Connection refused (error 111)
I20260806 06:30:52.534677 22480 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.12.188" port: 41667 } attempt: 1
W20260806 06:30:52.534888 22597 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260806 06:30:52.535145 22597 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
I20260806 06:30:52.535148 22483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:52.535024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.188:41667 (local address 127.0.0.1:47612)
0806 06:30:52.535080 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0806 06:30:52.535120 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.188:41667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260806 06:30:52.535183 22597 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 06:30:52.535215 22597 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260806 06:30:52.535478 22483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:52.535404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.188:41667 (local address 127.0.0.1:47624)
0806 06:30:52.535440 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0806 06:30:52.535463 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.188:41667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260806 06:30:52.535566 22480 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.12.188:41667: Network error: Client connection negotiation failed: client connection to 127.21.12.188:41667: connect: Connection refused (error 111)
W20260806 06:30:52.536468 22597 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 06:30:52.536514 22597 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 06:30:52.536533 22597 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 06:30:52.536550 22597 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 06:30:52.536574 22597 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 06:30:52.538098 22597 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.12.148:33149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785997581049705-21554-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785997581049705-21554-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.21.12.190:44049,127.21.12.189:40089,127.21.12.188:41667
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785997581049705-21554-0/krb5kdc/kudu_127.21.12.188.keytab
--principal=kudu/127.21.12.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.12.188:41667
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785997581049705-21554-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.21.12.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22597
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785997581049705-21554-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 06 Aug 2026 05:11:02 UTC on ccycloud.quasar-hvzbbz.root.comops.site
build id 4569617
I20260806 06:30:52.538364 22597 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 06:30:52.538579 22597 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 06:30:52.543150 22606 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:30:52.543150 22603 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 06:30:52.543186 22604 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 06:30:52.543339 22597 server_base.cc:1034] running on GCE node
I20260806 06:30:52.543532 22597 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 06:30:52.543784 22597 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 06:30:52.544929 22597 hybrid_clock.cc:648] HybridClock initialized: now 1785997852544913 us; error 26 us; skew 500 ppm
Aug 06 06:30:52 dist-test-slave-4tdj krb5kdc[22378](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997852, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 06:30:52.547232 22597 init.cc:377] Logged in from keytab as kudu/127.21.12.188@KRBTEST.COM (short username kudu)
I20260806 06:30:52.547633 22597 webserver.cc:460] Webserver started at http://127.21.12.188:33911/ using document root <none> and password file <none>
I20260806 06:30:52.547856 22597 fs_manager.cc:359] Metadata directory not provided
I20260806 06:30:52.547904 22597 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 06:30:52.548025 22597 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 06:30:52.549018 22597 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785997581049705-21554-0/minicluster-data/master-2/data/instance:
uuid: "8650c022f46d45c1892693114473bf4b"
format_stamp: "Formatted at 2026-08-06 06:30:52 on dist-test-slave-4tdj"
server_key: "11eeaf73fe28e35fa55aeedfeda55d8f"
server_key_iv: "94e6c6633f14f40912d89ed6082b850b"
server_key_version: "encryptionkey@0"
I20260806 06:30:52.549333 22597 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785997581049705-21554-0/minicluster-data/master-2/wal/instance:
uuid: "8650c022f46d45c1892693114473bf4b"
format_stamp: "Formatted at 2026-08-06 06:30:52 on dist-test-slave-4tdj"
server_key: "11eeaf73fe28e35fa55aeedfeda55d8f"
server_key_iv: "94e6c6633f14f40912d89ed6082b850b"
server_key_version: "encryptionkey@0"
I20260806 06:30:52.550634 22597 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260806 06:30:52.551318 22617 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 06:30:52.551468 22597 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260806 06:30:52.551551 22597 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785997581049705-21554-0/minicluster-data/master-2/data,/tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785997581049705-21554-0/minicluster-data/master-2/wal
uuid: "8650c022f46d45c1892693114473bf4b"
format_stamp: "Formatted at 2026-08-06 06:30:52 on dist-test-slave-4tdj"
server_key: "11eeaf73fe28e35fa55aeedfeda55d8f"
server_key_iv: "94e6c6633f14f40912d89ed6082b850b"
server_key_version: "encryptionkey@0"
I20260806 06:30:52.551608 22597 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785997581049705-21554-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785997581049705-21554-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785997581049705-21554-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 06:30:52.559903 22586 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.12.188" port: 41667 } attempt: 2
I20260806 06:30:52.560268 22589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:52.560158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.188:41667 (local address 127.0.0.1:47626)
0806 06:30:52.560209 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0806 06:30:52.560237 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.188:41667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260806 06:30:52.560590 22589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:52.560506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.188:41667 (local address 127.0.0.1:47642)
0806 06:30:52.560546 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0806 06:30:52.560569 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.21.12.188:41667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260806 06:30:52.560676 22586 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.21.12.188:41667: Network error: Client connection negotiation failed: client connection to 127.21.12.188:41667: connect: Connection refused (error 111)
I20260806 06:30:52.561877 22597 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 06:30:52.563525 22597 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 06:30:52.563663 22597 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 06:30:52.567315 22597 rpc_server.cc:307] RPC server started. Bound to: 127.21.12.188:41667
I20260806 06:30:52.567350 22677 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.12.188:41667 every 8 connection(s)
I20260806 06:30:52.567646 22597 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785997581049705-21554-0/minicluster-data/master-2/data/info.pb
I20260806 06:30:52.568564 22678 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 06:30:52.569998 22678 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.12.190" port: 44049 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 06:30:52 dist-test-slave-4tdj krb5kdc[22378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997852, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.188@KRBTEST.COM for kudu/127.21.12.190@KRBTEST.COM
I20260806 06:30:52.577797 21554 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHhLhT/build/release/bin/kudu as pid 22597
I20260806 06:30:52.577920 21554 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHhLhT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785997581049705-21554-0/minicluster-data/master-2/wal/instance
I20260806 06:30:52.577977 22484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:52.570629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34994 (local address 127.21.12.190:44049)
0806 06:30:52.570683 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0806 06:30:52.570685 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:30:52.570852 (+ 167us) server_negotiation.cc:407] Connection header received
0806 06:30:52.571320 (+ 468us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:30:52.571321 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:30:52.571335 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:30:52.571386 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:30:52.571965 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:52.572444 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:52.573366 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:52.573562 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:52.573618 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:30:52.575640 (+ 2022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:30:52.575642 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:30:52.575642 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 06:30:52.575650 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:30:52.576702 (+ 1052us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:30:52.577053 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:30:52.577054 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:30:52.577054 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:30:52.577119 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:30:52.577331 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:30:52.577332 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:30:52.577332 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:30:52.577441 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:30:52.577495 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0806 06:30:52.577804 (+ 309us) server_negotiation.cc:299] Negotiation successful
0806 06:30:52.577885 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260806 06:30:52.578141 21554 external_mini_cluster.cc:1428] Setting key 3bc48559d402c9758f70c4f5c78f77a5
I20260806 06:30:52.578203 22680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:52.570589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.190:44049 (local address 127.0.0.1:34994)
0806 06:30:52.570743 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0806 06:30:52.570781 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0806 06:30:52.571224 (+ 443us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:30:52.571451 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:30:52.571455 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:30:52.571700 (+ 245us) client_negotiation.cc:189] Negotiated authn=SASL
0806 06:30:52.571881 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:30:52.571885 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:30:52.572524 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:30:52.572525 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:30:52.573285 (+ 760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:30:52.573287 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:30:52.573657 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:30:52.573658 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:30:52.573769 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:30:52.574219 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 06:30:52.574228 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0806 06:30:52.575548 (+ 1320us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 06:30:52.576773 (+ 1225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:30:52.576780 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:30:52.576782 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:30:52.576984 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:30:52.577173 (+ 189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:30:52.577174 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:30:52.577174 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:30:52.577273 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:30:52.577495 (+ 222us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 06:30:52.577503 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 06:30:52.577651 (+ 148us) client_negotiation.cc:769] Sending connection context
0806 06:30:52.577766 (+ 115us) client_negotiation.cc:240] Negotiation successful
0806 06:30:52.577927 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":78,"threads_started":1}
I20260806 06:30:52.578830 22678 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.12.189" port: 40089 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 06:30:52 dist-test-slave-4tdj krb5kdc[22378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997852, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.188@KRBTEST.COM for kudu/127.21.12.189@KRBTEST.COM
Aug 06 06:30:52 dist-test-slave-4tdj krb5kdc[22378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997852, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.12.188@KRBTEST.COM
I20260806 06:30:52.585179 22680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:52.579047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.189:40089 (local address 127.0.0.1:56010)
0806 06:30:52.579090 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0806 06:30:52.579100 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0806 06:30:52.579134 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:30:52.579262 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:30:52.579263 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:30:52.579383 (+ 120us) client_negotiation.cc:189] Negotiated authn=SASL
0806 06:30:52.579429 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:30:52.579430 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:30:52.580094 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:30:52.580095 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:30:52.580599 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:30:52.580602 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:30:52.581026 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:30:52.581027 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:30:52.581100 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:30:52.581459 (+ 359us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 06:30:52.581463 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0806 06:30:52.582823 (+ 1360us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 06:30:52.584314 (+ 1491us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:30:52.584314 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:30:52.584315 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:30:52.584442 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:30:52.584614 (+ 172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:30:52.584615 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:30:52.584615 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:30:52.584679 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:30:52.584887 (+ 208us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 06:30:52.584888 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 06:30:52.584932 (+ 44us) client_negotiation.cc:769] Sending connection context
0806 06:30:52.584990 (+ 58us) client_negotiation.cc:240] Negotiation successful
0806 06:30:52.585081 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22}
I20260806 06:30:52.585177 22591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:52.579127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56010 (local address 127.21.12.189:40089)
0806 06:30:52.579169 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0806 06:30:52.579171 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:30:52.579178 (+ 7us) server_negotiation.cc:407] Connection header received
0806 06:30:52.579200 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:30:52.579201 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:30:52.579210 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:30:52.579254 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:30:52.579531 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:52.580018 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:52.580675 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:52.580971 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:52.581018 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:30:52.582899 (+ 1881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:30:52.582901 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:30:52.582901 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 06:30:52.582909 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:30:52.584248 (+ 1339us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:30:52.584495 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:30:52.584496 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:30:52.584497 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:30:52.584546 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:30:52.584730 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:30:52.584731 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:30:52.584731 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:30:52.584835 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:30:52.584888 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0806 06:30:52.585019 (+ 131us) server_negotiation.cc:299] Negotiation successful
0806 06:30:52.585080 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260806 06:30:52.585556 22678 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.21.12.188" port: 41667 } has no permanent_uuid. Determining permanent_uuid...
I20260806 06:30:52.586324 22684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:52.578503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47652 (local address 127.21.12.188:41667)
0806 06:30:52.578724 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0806 06:30:52.578728 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 06:30:52.578747 (+ 19us) server_negotiation.cc:407] Connection header received
0806 06:30:52.578766 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:30:52.578768 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:30:52.578781 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:30:52.578847 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:30:52.579449 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:52.580131 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:52.580898 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:52.581283 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:52.581342 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:30:52.583604 (+ 2262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:30:52.583612 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:30:52.583612 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 06:30:52.583619 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:30:52.585016 (+ 1397us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:30:52.585375 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:30:52.585379 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:30:52.585380 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:30:52.585439 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:30:52.585636 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:30:52.585637 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:30:52.585637 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:30:52.585791 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:30:52.585925 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0806 06:30:52.586107 (+ 182us) server_negotiation.cc:299] Negotiation successful
0806 06:30:52.586209 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":192,"thread_start_us":75,"threads_started":1}
Aug 06 06:30:52 dist-test-slave-4tdj krb5kdc[22378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997852, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.188@KRBTEST.COM for kudu/127.21.12.188@KRBTEST.COM
Aug 06 06:30:52 dist-test-slave-4tdj krb5kdc[22378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997852, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.12.190@KRBTEST.COM
I20260806 06:30:52.592410 22680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:52.585765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.188:41667 (local address 127.0.0.1:47664)
0806 06:30:52.585812 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0806 06:30:52.585822 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0806 06:30:52.585854 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:30:52.586046 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:30:52.586047 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:30:52.586214 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0806 06:30:52.586248 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:30:52.586249 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:30:52.587223 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:30:52.587224 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:30:52.587675 (+ 451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:30:52.587676 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:30:52.588003 (+ 327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:30:52.588004 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:30:52.588047 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:30:52.588391 (+ 344us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 06:30:52.588395 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0806 06:30:52.589479 (+ 1084us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 06:30:52.591548 (+ 2069us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:30:52.591549 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:30:52.591549 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:30:52.591708 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:30:52.591877 (+ 169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:30:52.591877 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:30:52.591878 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:30:52.591931 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:30:52.592128 (+ 197us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 06:30:52.592129 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 06:30:52.592165 (+ 36us) client_negotiation.cc:769] Sending connection context
0806 06:30:52.592225 (+ 60us) client_negotiation.cc:240] Negotiation successful
0806 06:30:52.592315 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260806 06:30:52.592482 22686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:52.585805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47664 (local address 127.21.12.188:41667)
0806 06:30:52.585942 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0806 06:30:52.585943 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0806 06:30:52.585950 (+ 7us) server_negotiation.cc:407] Connection header received
0806 06:30:52.585967 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:30:52.585967 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:30:52.585975 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:30:52.586025 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:30:52.586334 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:52.587158 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:52.587737 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:52.587951 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:52.588000 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:30:52.589554 (+ 1554us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:30:52.589555 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:30:52.589555 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 06:30:52.589561 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:30:52.591476 (+ 1915us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:30:52.591776 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:30:52.591777 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:30:52.591778 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:30:52.591826 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:30:52.591978 (+ 152us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:30:52.591979 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:30:52.591980 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:30:52.592071 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:30:52.592128 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0806 06:30:52.592250 (+ 122us) server_negotiation.cc:299] Negotiation successful
0806 06:30:52.592364 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":50,"server-negotiator.queue_time_us":115,"thread_start_us":79,"threads_started":1}
I20260806 06:30:52.593315 22484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:52.587018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35010 (local address 127.21.12.190:44049)
0806 06:30:52.587062 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0806 06:30:52.587065 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 06:30:52.587083 (+ 18us) server_negotiation.cc:407] Connection header received
0806 06:30:52.587102 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:30:52.587103 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:30:52.587111 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:30:52.587151 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:30:52.587549 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:52.587981 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:52.588711 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:52.588927 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:52.588988 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:30:52.590939 (+ 1951us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:30:52.590941 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:30:52.590941 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 06:30:52.590948 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:30:52.592229 (+ 1281us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:30:52.592548 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:30:52.592549 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:30:52.592550 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:30:52.592609 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:30:52.592829 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:30:52.592830 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:30:52.592830 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:30:52.592952 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:30:52.593126 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0806 06:30:52.593166 (+ 40us) server_negotiation.cc:299] Negotiation successful
0806 06:30:52.593236 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260806 06:30:52.594170 22678 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8650c022f46d45c1892693114473bf4b: Bootstrap starting.
I20260806 06:30:52.594875 22678 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8650c022f46d45c1892693114473bf4b: Neither blocks nor log segments found. Creating new log.
I20260806 06:30:52.595160 22678 log.cc:824] T 00000000000000000000000000000000 P 8650c022f46d45c1892693114473bf4b: Log is configured to *not* fsync() on all Append() calls
I20260806 06:30:52.595882 22678 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8650c022f46d45c1892693114473bf4b: No bootstrap required, opened a new log
I20260806 06:30:52.597371 22678 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8650c022f46d45c1892693114473bf4b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3179d6681ac24153ab6385dbba22d241" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 44049 } } peers { permanent_uuid: "0350b17ccfb24e5c82c3a2931bbb5caf" member_type: VOTER last_known_addr { host: "127.21.12.189" port: 40089 } } peers { permanent_uuid: "8650c022f46d45c1892693114473bf4b" member_type: VOTER last_known_addr { host: "127.21.12.188" port: 41667 } }
I20260806 06:30:52.597496 22678 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8650c022f46d45c1892693114473bf4b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8650c022f46d45c1892693114473bf4b, State: Initialized, Role: FOLLOWER
I20260806 06:30:52.597572 22678 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8650c022f46d45c1892693114473bf4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3179d6681ac24153ab6385dbba22d241" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 44049 } } peers { permanent_uuid: "0350b17ccfb24e5c82c3a2931bbb5caf" member_type: VOTER last_known_addr { host: "127.21.12.189" port: 40089 } } peers { permanent_uuid: "8650c022f46d45c1892693114473bf4b" member_type: VOTER last_known_addr { host: "127.21.12.188" port: 41667 } }
Aug 06 06:30:52 dist-test-slave-4tdj krb5kdc[22378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997852, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.21.12.189@KRBTEST.COM
I20260806 06:30:52.597827 22688 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8650c022f46d45c1892693114473bf4b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3179d6681ac24153ab6385dbba22d241" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 44049 } } peers { permanent_uuid: "0350b17ccfb24e5c82c3a2931bbb5caf" member_type: VOTER last_known_addr { host: "127.21.12.189" port: 40089 } } peers { permanent_uuid: "8650c022f46d45c1892693114473bf4b" member_type: VOTER last_known_addr { host: "127.21.12.188" port: 41667 } } }
I20260806 06:30:52.597885 22678 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8650c022f46d45c1892693114473bf4b [sys.catalog]: configured and running, proceeding with master startup.
I20260806 06:30:52.597913 22688 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8650c022f46d45c1892693114473bf4b [sys.catalog]: This master's current role is: FOLLOWER
W20260806 06:30:52.599654 22706 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8650c022f46d45c1892693114473bf4b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 06:30:52.599756 22706 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 06:30:52.600780 22591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:52.593948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56012 (local address 127.21.12.189:40089)
0806 06:30:52.593992 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0806 06:30:52.593994 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:30:52.594002 (+ 8us) server_negotiation.cc:407] Connection header received
0806 06:30:52.594024 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:30:52.594025 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:30:52.594034 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:30:52.594114 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:30:52.594578 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:52.595228 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:52.595939 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:52.596225 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:52.596298 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:30:52.598313 (+ 2015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:30:52.598315 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:30:52.598316 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 06:30:52.598322 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:30:52.599778 (+ 1456us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:30:52.600119 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:30:52.600121 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:30:52.600121 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:30:52.600181 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:30:52.600340 (+ 159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:30:52.600341 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:30:52.600341 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:30:52.600444 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:30:52.600596 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0806 06:30:52.600631 (+ 35us) server_negotiation.cc:299] Negotiation successful
0806 06:30:52.600695 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260806 06:30:52.601941 21554 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 06:30:52.599983 22597 master_runner.cc:186] Error getting master registration for 127.21.12.190:44049: OK, instance_id { permanent_uuid: "3179d6681ac24153ab6385dbba22d241" instance_seqno: 1785997852270523 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260806 06:30:52.602118 22597 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260806 06:30:52.602936 22448 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "3179d6681ac24153ab6385dbba22d241"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35010
I20260806 06:30:52.610682 22480 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.12.188" port: 41667 } attempt: 2
Aug 06 06:30:52 dist-test-slave-4tdj krb5kdc[22378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997852, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.190@KRBTEST.COM for kudu/127.21.12.188@KRBTEST.COM
I20260806 06:30:52.618098 22483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:52.610922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.188:41667 (local address 127.0.0.1:47668)
0806 06:30:52.610978 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0806 06:30:52.610991 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 06:30:52.611042 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:30:52.611179 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:30:52.611180 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:30:52.611346 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0806 06:30:52.611393 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:30:52.611395 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:30:52.612268 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:30:52.612269 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:30:52.613085 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:30:52.613088 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:30:52.613531 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:30:52.613532 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:30:52.613588 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:30:52.614006 (+ 418us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 06:30:52.614011 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0806 06:30:52.615430 (+ 1419us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 06:30:52.617093 (+ 1663us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:30:52.617094 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:30:52.617095 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:30:52.617283 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:30:52.617452 (+ 169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:30:52.617453 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:30:52.617453 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:30:52.617508 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:30:52.617789 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 06:30:52.617790 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 06:30:52.617843 (+ 53us) client_negotiation.cc:769] Sending connection context
0806 06:30:52.617905 (+ 62us) client_negotiation.cc:240] Negotiation successful
0806 06:30:52.617991 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260806 06:30:52.618100 22686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:52.611002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47668 (local address 127.21.12.188:41667)
0806 06:30:52.611050 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0806 06:30:52.611052 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:30:52.611080 (+ 28us) server_negotiation.cc:407] Connection header received
0806 06:30:52.611108 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:30:52.611109 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:30:52.611118 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:30:52.611163 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:30:52.611463 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:52.612201 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:52.613163 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:52.613471 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:52.613525 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:30:52.615505 (+ 1980us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:30:52.615506 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:30:52.615507 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 06:30:52.615513 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:30:52.616996 (+ 1483us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:30:52.617347 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:30:52.617348 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:30:52.617348 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:30:52.617401 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:30:52.617557 (+ 156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:30:52.617558 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:30:52.617558 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:30:52.617728 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:30:52.617796 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0806 06:30:52.617934 (+ 138us) server_negotiation.cc:299] Negotiation successful
0806 06:30:52.618011 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260806 06:30:52.619838 22480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3179d6681ac24153ab6385dbba22d241: Bootstrap starting.
I20260806 06:30:52.620574 22480 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3179d6681ac24153ab6385dbba22d241: Neither blocks nor log segments found. Creating new log.
I20260806 06:30:52.620872 22480 log.cc:824] T 00000000000000000000000000000000 P 3179d6681ac24153ab6385dbba22d241: Log is configured to *not* fsync() on all Append() calls
I20260806 06:30:52.621654 22480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3179d6681ac24153ab6385dbba22d241: No bootstrap required, opened a new log
I20260806 06:30:52.622992 22480 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3179d6681ac24153ab6385dbba22d241 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3179d6681ac24153ab6385dbba22d241" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 44049 } } peers { permanent_uuid: "0350b17ccfb24e5c82c3a2931bbb5caf" member_type: VOTER last_known_addr { host: "127.21.12.189" port: 40089 } } peers { permanent_uuid: "8650c022f46d45c1892693114473bf4b" member_type: VOTER last_known_addr { host: "127.21.12.188" port: 41667 } }
I20260806 06:30:52.623135 22480 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3179d6681ac24153ab6385dbba22d241 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3179d6681ac24153ab6385dbba22d241, State: Initialized, Role: FOLLOWER
I20260806 06:30:52.623219 22480 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3179d6681ac24153ab6385dbba22d241 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3179d6681ac24153ab6385dbba22d241" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 44049 } } peers { permanent_uuid: "0350b17ccfb24e5c82c3a2931bbb5caf" member_type: VOTER last_known_addr { host: "127.21.12.189" port: 40089 } } peers { permanent_uuid: "8650c022f46d45c1892693114473bf4b" member_type: VOTER last_known_addr { host: "127.21.12.188" port: 41667 } }
I20260806 06:30:52.623534 22480 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3179d6681ac24153ab6385dbba22d241 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 06:30:52.623574 22448 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3179d6681ac24153ab6385dbba22d241 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 06:30:52.623624 22448 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3179d6681ac24153ab6385dbba22d241 [term 0 FOLLOWER]: Advancing to term 1
I20260806 06:30:52.624366 22448 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3179d6681ac24153ab6385dbba22d241 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3179d6681ac24153ab6385dbba22d241" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 44049 } } peers { permanent_uuid: "0350b17ccfb24e5c82c3a2931bbb5caf" member_type: VOTER last_known_addr { host: "127.21.12.189" port: 40089 } } peers { permanent_uuid: "8650c022f46d45c1892693114473bf4b" member_type: VOTER last_known_addr { host: "127.21.12.188" port: 41667 } }
I20260806 06:30:52.624621 22448 leader_election.cc:288] T 00000000000000000000000000000000 P 3179d6681ac24153ab6385dbba22d241 [CANDIDATE]: Term 1 election: Requested vote from peers 0350b17ccfb24e5c82c3a2931bbb5caf (127.21.12.189:40089), 8650c022f46d45c1892693114473bf4b (127.21.12.188:41667)
I20260806 06:30:52.625094 22646 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3179d6681ac24153ab6385dbba22d241" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "8650c022f46d45c1892693114473bf4b"
I20260806 06:30:52.625232 22646 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8650c022f46d45c1892693114473bf4b [term 0 FOLLOWER]: Advancing to term 1
W20260806 06:30:52.625381 22726 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3179d6681ac24153ab6385dbba22d241: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 06:30:52.625430 22726 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 06:30:52.626111 22646 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8650c022f46d45c1892693114473bf4b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3179d6681ac24153ab6385dbba22d241 in term 1.
I20260806 06:30:52.626489 22417 leader_election.cc:302] T 00000000000000000000000000000000 P 3179d6681ac24153ab6385dbba22d241 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3179d6681ac24153ab6385dbba22d241, 8650c022f46d45c1892693114473bf4b; no voters:
I20260806 06:30:52.627007 22728 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3179d6681ac24153ab6385dbba22d241 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 06:30:52.627192 22728 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3179d6681ac24153ab6385dbba22d241 [term 1 LEADER]: Becoming Leader. State: Replica: 3179d6681ac24153ab6385dbba22d241, State: Running, Role: LEADER
I20260806 06:30:52.628018 22728 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3179d6681ac24153ab6385dbba22d241 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3179d6681ac24153ab6385dbba22d241" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 44049 } } peers { permanent_uuid: "0350b17ccfb24e5c82c3a2931bbb5caf" member_type: VOTER last_known_addr { host: "127.21.12.189" port: 40089 } } peers { permanent_uuid: "8650c022f46d45c1892693114473bf4b" member_type: VOTER last_known_addr { host: "127.21.12.188" port: 41667 } }
I20260806 06:30:52.628535 22728 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3179d6681ac24153ab6385dbba22d241 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3179d6681ac24153ab6385dbba22d241. Latest consensus state: current_term: 1 leader_uuid: "3179d6681ac24153ab6385dbba22d241" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3179d6681ac24153ab6385dbba22d241" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 44049 } } peers { permanent_uuid: "0350b17ccfb24e5c82c3a2931bbb5caf" member_type: VOTER last_known_addr { host: "127.21.12.189" port: 40089 } } peers { permanent_uuid: "8650c022f46d45c1892693114473bf4b" member_type: VOTER last_known_addr { host: "127.21.12.188" port: 41667 } } }
I20260806 06:30:52.628634 22728 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3179d6681ac24153ab6385dbba22d241 [sys.catalog]: This master's current role is: LEADER
I20260806 06:30:52.623507 22709 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3179d6681ac24153ab6385dbba22d241 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3179d6681ac24153ab6385dbba22d241" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 44049 } } peers { permanent_uuid: "0350b17ccfb24e5c82c3a2931bbb5caf" member_type: VOTER last_known_addr { host: "127.21.12.189" port: 40089 } } peers { permanent_uuid: "8650c022f46d45c1892693114473bf4b" member_type: VOTER last_known_addr { host: "127.21.12.188" port: 41667 } } }
I20260806 06:30:52.628858 22709 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3179d6681ac24153ab6385dbba22d241 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 06:30:52.627431 22394 master_runner.cc:186] Error getting master registration for 127.21.12.189:40089: OK, instance_id { permanent_uuid: "0350b17ccfb24e5c82c3a2931bbb5caf" instance_seqno: 1785997852418746 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260806 06:30:52.629832 22394 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260806 06:30:52.630118 22735 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 06:30:52.630395 22735 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 06:30:52.631814 22646 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8650c022f46d45c1892693114473bf4b [term 1 FOLLOWER]: Refusing update from remote peer 3179d6681ac24153ab6385dbba22d241: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 06:30:52.632097 22709 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3179d6681ac24153ab6385dbba22d241 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8650c022f46d45c1892693114473bf4b" member_type: VOTER last_known_addr { host: "127.21.12.188" port: 41667 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 06:30:52.633311 22688 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8650c022f46d45c1892693114473bf4b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3179d6681ac24153ab6385dbba22d241. Latest consensus state: current_term: 1 leader_uuid: "3179d6681ac24153ab6385dbba22d241" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3179d6681ac24153ab6385dbba22d241" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 44049 } } peers { permanent_uuid: "0350b17ccfb24e5c82c3a2931bbb5caf" member_type: VOTER last_known_addr { host: "127.21.12.189" port: 40089 } } peers { permanent_uuid: "8650c022f46d45c1892693114473bf4b" member_type: VOTER last_known_addr { host: "127.21.12.188" port: 41667 } } }
I20260806 06:30:52.633400 22688 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8650c022f46d45c1892693114473bf4b [sys.catalog]: This master's current role is: FOLLOWER
I20260806 06:30:52.635726 22728 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3179d6681ac24153ab6385dbba22d241 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3179d6681ac24153ab6385dbba22d241" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3179d6681ac24153ab6385dbba22d241" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 44049 } } peers { permanent_uuid: "0350b17ccfb24e5c82c3a2931bbb5caf" member_type: VOTER last_known_addr { host: "127.21.12.189" port: 40089 } } peers { permanent_uuid: "8650c022f46d45c1892693114473bf4b" member_type: VOTER last_known_addr { host: "127.21.12.188" port: 41667 } } }
I20260806 06:30:52.635802 22728 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3179d6681ac24153ab6385dbba22d241 [sys.catalog]: This master's current role is: LEADER
I20260806 06:30:52.636236 22688 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8650c022f46d45c1892693114473bf4b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3179d6681ac24153ab6385dbba22d241" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3179d6681ac24153ab6385dbba22d241" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 44049 } } peers { permanent_uuid: "0350b17ccfb24e5c82c3a2931bbb5caf" member_type: VOTER last_known_addr { host: "127.21.12.189" port: 40089 } } peers { permanent_uuid: "8650c022f46d45c1892693114473bf4b" member_type: VOTER last_known_addr { host: "127.21.12.188" port: 41667 } } }
I20260806 06:30:52.636319 22688 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8650c022f46d45c1892693114473bf4b [sys.catalog]: This master's current role is: FOLLOWER
I20260806 06:30:52.636619 22738 mvcc.cc:205] Tried to move back new op lower bound from 7315447204377604096 to 7315447204365541376. Current Snapshot: MvccSnapshot[applied={T|T < 7315447204377604096}]
I20260806 06:30:52.636785 22742 mvcc.cc:205] Tried to move back new op lower bound from 7315447204377604096 to 7315447204365541376. Current Snapshot: MvccSnapshot[applied={T|T < 7315447204377604096}]
I20260806 06:30:52.637558 22735 catalog_manager.cc:1347] Generated new cluster ID: 022faf664df04188abd49e1508f67d61
I20260806 06:30:52.637601 22735 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 06:30:52.645900 22591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:52.630270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56016 (local address 127.21.12.189:40089)
0806 06:30:52.630330 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0806 06:30:52.630332 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:30:52.631717 (+ 1385us) server_negotiation.cc:407] Connection header received
0806 06:30:52.631837 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:30:52.631838 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:30:52.631849 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:30:52.631879 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:30:52.632754 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:52.633462 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:52.636792 (+ 3330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:52.637095 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:52.637143 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:30:52.642278 (+ 5135us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:30:52.642279 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:30:52.642280 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 06:30:52.642317 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:30:52.644055 (+ 1738us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:30:52.644432 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:30:52.644433 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:30:52.644434 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:30:52.644491 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:30:52.644974 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:30:52.644975 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:30:52.644975 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:30:52.645115 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:30:52.645224 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0806 06:30:52.645723 (+ 499us) server_negotiation.cc:299] Negotiation successful
0806 06:30:52.645804 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260806 06:30:52.648339 22686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:52.631171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47682 (local address 127.21.12.188:41667)
0806 06:30:52.631221 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0806 06:30:52.631223 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:30:52.633920 (+ 2697us) server_negotiation.cc:407] Connection header received
0806 06:30:52.634023 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:30:52.634024 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:30:52.634035 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:30:52.634098 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:30:52.635140 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:52.635856 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:52.639359 (+ 3503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:52.639634 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:52.639674 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:30:52.642605 (+ 2931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:30:52.642607 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:30:52.642607 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 06:30:52.642614 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:30:52.644218 (+ 1604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:30:52.644957 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:30:52.644958 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:30:52.644959 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:30:52.645018 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:30:52.647435 (+ 2417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:30:52.647436 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:30:52.647436 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:30:52.647580 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:30:52.647670 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0806 06:30:52.648174 (+ 504us) server_negotiation.cc:299] Negotiation successful
0806 06:30:52.648256 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260806 06:30:52.648854 22484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:52.631443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35020 (local address 127.21.12.190:44049)
0806 06:30:52.631478 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0806 06:30:52.631479 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0806 06:30:52.634120 (+ 2641us) server_negotiation.cc:407] Connection header received
0806 06:30:52.634221 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:30:52.634222 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:30:52.634232 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:30:52.634281 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:30:52.635220 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:52.635891 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:52.639984 (+ 4093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:52.640273 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:52.640315 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:30:52.645417 (+ 5102us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:30:52.645418 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:30:52.645419 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 06:30:52.645426 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:30:52.647177 (+ 1751us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:30:52.647694 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:30:52.647695 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:30:52.647696 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:30:52.647754 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:30:52.648035 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:30:52.648036 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:30:52.648037 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:30:52.648192 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:30:52.648276 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0806 06:30:52.648703 (+ 427us) server_negotiation.cc:299] Negotiation successful
0806 06:30:52.648772 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260806 06:30:52.651795 22586 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.21.12.188" port: 41667 } attempt: 3
Aug 06 06:30:52 dist-test-slave-4tdj krb5kdc[22378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785997852, etypes {rep=17 tkt=17 ses=17}, kudu/127.21.12.189@KRBTEST.COM for kudu/127.21.12.188@KRBTEST.COM
I20260806 06:30:52.659731 22589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:52.652051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.12.188:41667 (local address 127.0.0.1:47688)
0806 06:30:52.652110 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0806 06:30:52.652120 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0806 06:30:52.652161 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 06:30:52.652333 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 06:30:52.652334 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 06:30:52.652483 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0806 06:30:52.652524 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:30:52.652525 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:30:52.653392 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:30:52.653393 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:30:52.654223 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 06:30:52.654226 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 06:30:52.654871 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 06:30:52.654872 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 06:30:52.654929 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:30:52.655310 (+ 381us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 06:30:52.655315 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0806 06:30:52.656873 (+ 1558us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 06:30:52.658634 (+ 1761us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:30:52.658635 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:30:52.658635 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:30:52.658832 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:30:52.659037 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 06:30:52.659038 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 06:30:52.659038 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0806 06:30:52.659107 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 06:30:52.659422 (+ 315us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 06:30:52.659423 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 06:30:52.659481 (+ 58us) client_negotiation.cc:769] Sending connection context
0806 06:30:52.659538 (+ 57us) client_negotiation.cc:240] Negotiation successful
0806 06:30:52.659631 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260806 06:30:52.659987 22735 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 06:30:52.660207 22686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:52.652067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47688 (local address 127.21.12.188:41667)
0806 06:30:52.652221 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0806 06:30:52.652223 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:30:52.652229 (+ 6us) server_negotiation.cc:407] Connection header received
0806 06:30:52.652246 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:30:52.652247 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:30:52.652256 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:30:52.652297 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:30:52.652600 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:52.653284 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:52.654422 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:52.654722 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:52.654763 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:30:52.656990 (+ 2227us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 06:30:52.656991 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 06:30:52.656991 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 06:30:52.656997 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0806 06:30:52.658543 (+ 1546us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:30:52.658911 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:30:52.658912 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:30:52.658912 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:30:52.658978 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 06:30:52.659188 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 06:30:52.659189 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 06:30:52.659190 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 06:30:52.659324 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 06:30:52.659378 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0806 06:30:52.660029 (+ 651us) server_negotiation.cc:299] Negotiation successful
0806 06:30:52.660120 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136}
I20260806 06:30:52.660714 22735 catalog_manager.cc:1504] Loading token signing keys...
I20260806 06:30:52.661923 22586 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0350b17ccfb24e5c82c3a2931bbb5caf: Bootstrap starting.
I20260806 06:30:52.662672 22586 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0350b17ccfb24e5c82c3a2931bbb5caf: Neither blocks nor log segments found. Creating new log.
I20260806 06:30:52.663012 22586 log.cc:824] T 00000000000000000000000000000000 P 0350b17ccfb24e5c82c3a2931bbb5caf: Log is configured to *not* fsync() on all Append() calls
I20260806 06:30:52.668892 22735 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3179d6681ac24153ab6385dbba22d241: Generated new TSK 0
I20260806 06:30:52.669176 22735 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 06:30:52.672833 22586 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0350b17ccfb24e5c82c3a2931bbb5caf: No bootstrap required, opened a new log
I20260806 06:30:52.674847 22586 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0350b17ccfb24e5c82c3a2931bbb5caf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3179d6681ac24153ab6385dbba22d241" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 44049 } } peers { permanent_uuid: "0350b17ccfb24e5c82c3a2931bbb5caf" member_type: VOTER last_known_addr { host: "127.21.12.189" port: 40089 } } peers { permanent_uuid: "8650c022f46d45c1892693114473bf4b" member_type: VOTER last_known_addr { host: "127.21.12.188" port: 41667 } }
I20260806 06:30:52.674978 22586 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0350b17ccfb24e5c82c3a2931bbb5caf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0350b17ccfb24e5c82c3a2931bbb5caf, State: Initialized, Role: FOLLOWER
I20260806 06:30:52.675050 22586 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0350b17ccfb24e5c82c3a2931bbb5caf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3179d6681ac24153ab6385dbba22d241" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 44049 } } peers { permanent_uuid: "0350b17ccfb24e5c82c3a2931bbb5caf" member_type: VOTER last_known_addr { host: "127.21.12.189" port: 40089 } } peers { permanent_uuid: "8650c022f46d45c1892693114473bf4b" member_type: VOTER last_known_addr { host: "127.21.12.188" port: 41667 } }
I20260806 06:30:52.675401 22586 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0350b17ccfb24e5c82c3a2931bbb5caf [sys.catalog]: configured and running, proceeding with master startup.
I20260806 06:30:52.675490 22547 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0350b17ccfb24e5c82c3a2931bbb5caf [term 0 FOLLOWER]: Advancing to term 1
I20260806 06:30:52.676434 22547 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0350b17ccfb24e5c82c3a2931bbb5caf [term 1 FOLLOWER]: Refusing update from remote peer 3179d6681ac24153ab6385dbba22d241: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 06:30:52.676841 22709 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3179d6681ac24153ab6385dbba22d241 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0350b17ccfb24e5c82c3a2931bbb5caf" member_type: VOTER last_known_addr { host: "127.21.12.189" port: 40089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 06:30:52.676620 22549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3179d6681ac24153ab6385dbba22d241" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "0350b17ccfb24e5c82c3a2931bbb5caf"
I20260806 06:30:52.677706 22549 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0350b17ccfb24e5c82c3a2931bbb5caf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3179d6681ac24153ab6385dbba22d241 in term 1.
I20260806 06:30:52.677291 22749 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0350b17ccfb24e5c82c3a2931bbb5caf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3179d6681ac24153ab6385dbba22d241" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 44049 } } peers { permanent_uuid: "0350b17ccfb24e5c82c3a2931bbb5caf" member_type: VOTER last_known_addr { host: "127.21.12.189" port: 40089 } } peers { permanent_uuid: "8650c022f46d45c1892693114473bf4b" member_type: VOTER last_known_addr { host: "127.21.12.188" port: 41667 } } }
I20260806 06:30:52.677994 22749 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0350b17ccfb24e5c82c3a2931bbb5caf [sys.catalog]: This master's current role is: FOLLOWER
I20260806 06:30:52.680101 22749 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0350b17ccfb24e5c82c3a2931bbb5caf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3179d6681ac24153ab6385dbba22d241. Latest consensus state: current_term: 1 leader_uuid: "3179d6681ac24153ab6385dbba22d241" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3179d6681ac24153ab6385dbba22d241" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 44049 } } peers { permanent_uuid: "0350b17ccfb24e5c82c3a2931bbb5caf" member_type: VOTER last_known_addr { host: "127.21.12.189" port: 40089 } } peers { permanent_uuid: "8650c022f46d45c1892693114473bf4b" member_type: VOTER last_known_addr { host: "127.21.12.188" port: 41667 } } }
I20260806 06:30:52.680197 22749 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0350b17ccfb24e5c82c3a2931bbb5caf [sys.catalog]: This master's current role is: FOLLOWER
I20260806 06:30:52.680229 22749 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0350b17ccfb24e5c82c3a2931bbb5caf [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3179d6681ac24153ab6385dbba22d241" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3179d6681ac24153ab6385dbba22d241" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 44049 } } peers { permanent_uuid: "0350b17ccfb24e5c82c3a2931bbb5caf" member_type: VOTER last_known_addr { host: "127.21.12.189" port: 40089 } } peers { permanent_uuid: "8650c022f46d45c1892693114473bf4b" member_type: VOTER last_known_addr { host: "127.21.12.188" port: 41667 } } }
I20260806 06:30:52.680272 22749 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0350b17ccfb24e5c82c3a2931bbb5caf [sys.catalog]: This master's current role is: FOLLOWER
I20260806 06:30:52.680826 22766 mvcc.cc:205] Tried to move back new op lower bound from 7315447204527808512 to 7315447204365541376. Current Snapshot: MvccSnapshot[applied={T|T < 7315447204377604096}]
W20260806 06:30:52.681411 22765 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0350b17ccfb24e5c82c3a2931bbb5caf: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 06:30:52.681510 22765 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260806 06:30:52.702595 22591 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 06:30:52.700863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56024 (local address 127.21.12.189:40089)
0806 06:30:52.700909 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0806 06:30:52.700911 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:30:52.701860 (+ 949us) server_negotiation.cc:407] Connection header received
0806 06:30:52.701977 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:30:52.701978 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:30:52.701989 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:30:52.702018 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:30:52.702525 (+ 507us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56024: BlockingRecv error: recv got EOF from 127.0.0.1:56024 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260806 06:30:52.703518 22686 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 06:30:52.701296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47696 (local address 127.21.12.188:41667)
0806 06:30:52.701337 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0806 06:30:52.701339 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:30:52.703013 (+ 1674us) server_negotiation.cc:407] Connection header received
0806 06:30:52.703120 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:30:52.703121 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:30:52.703132 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:30:52.703160 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:30:52.703452 (+ 292us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47696: BlockingRecv error: recv got EOF from 127.0.0.1:47696 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260806 06:30:52.704433 22484 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 06:30:52.701609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35030 (local address 127.21.12.190:44049)
0806 06:30:52.701646 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0806 06:30:52.701648 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:30:52.703927 (+ 2279us) server_negotiation.cc:407] Connection header received
0806 06:30:52.704020 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:30:52.704021 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:30:52.704032 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:30:52.704059 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:30:52.704371 (+ 312us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35030: BlockingRecv error: recv got EOF from 127.0.0.1:35030 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260806 06:30:52.706898 22591 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 06:30:52.705726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56032 (local address 127.21.12.189:40089)
0806 06:30:52.705759 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0806 06:30:52.705761 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:30:52.706380 (+ 619us) server_negotiation.cc:407] Connection header received
0806 06:30:52.706487 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:30:52.706488 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:30:52.706507 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:30:52.706535 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:30:52.706851 (+ 316us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56032: BlockingRecv error: recv got EOF from 127.0.0.1:56032 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260806 06:30:52.709837 22591 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 06:30:52.709145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56048 (local address 127.21.12.189:40089)
0806 06:30:52.709180 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0806 06:30:52.709181 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0806 06:30:52.709402 (+ 221us) server_negotiation.cc:407] Connection header received
0806 06:30:52.709489 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:30:52.709490 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:30:52.709499 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:30:52.709525 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:30:52.709795 (+ 270us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56048: BlockingRecv error: recv got EOF from 127.0.0.1:56048 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260806 06:30:52.713605 22591 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 06:30:52.712894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56056 (local address 127.21.12.189:40089)
0806 06:30:52.712926 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0806 06:30:52.712928 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:30:52.713160 (+ 232us) server_negotiation.cc:407] Connection header received
0806 06:30:52.713247 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:30:52.713248 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:30:52.713258 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:30:52.713285 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:30:52.713564 (+ 279us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56056: BlockingRecv error: recv got EOF from 127.0.0.1:56056 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260806 06:30:52.719367 22591 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 06:30:52.718688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56068 (local address 127.21.12.189:40089)
0806 06:30:52.718718 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0806 06:30:52.718719 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0806 06:30:52.718929 (+ 210us) server_negotiation.cc:407] Connection header received
0806 06:30:52.719010 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:30:52.719011 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:30:52.719020 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:30:52.719045 (+ 25us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:30:52.719327 (+ 282us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56068: BlockingRecv error: recv got EOF from 127.0.0.1:56068 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
W20260806 06:30:52.729297 22591 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 06:30:52.728494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56074 (local address 127.21.12.189:40089)
0806 06:30:52.728530 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0806 06:30:52.728532 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:30:52.728762 (+ 230us) server_negotiation.cc:407] Connection header received
0806 06:30:52.728854 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:30:52.728856 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:30:52.728866 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:30:52.728894 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:30:52.729235 (+ 341us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56074: BlockingRecv error: recv got EOF from 127.0.0.1:56074 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260806 06:30:52.747874 22591 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 06:30:52.746882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56088 (local address 127.21.12.189:40089)
0806 06:30:52.746926 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0806 06:30:52.746928 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:30:52.747233 (+ 305us) server_negotiation.cc:407] Connection header received
0806 06:30:52.747340 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:30:52.747342 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:30:52.747354 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:30:52.747385 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:30:52.747791 (+ 406us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56088: BlockingRecv error: recv got EOF from 127.0.0.1:56088 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260806 06:30:52.782130 22591 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 06:30:52.781387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56104 (local address 127.21.12.189:40089)
0806 06:30:52.781436 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0806 06:30:52.781439 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 06:30:52.781532 (+ 93us) server_negotiation.cc:407] Connection header received
0806 06:30:52.781619 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:30:52.781621 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:30:52.781635 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:30:52.781700 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:30:52.782063 (+ 363us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56104: BlockingRecv error: recv got EOF from 127.0.0.1:56104 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
W20260806 06:30:52.847826 22591 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 06:30:52.847165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56118 (local address 127.21.12.189:40089)
0806 06:30:52.847222 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0806 06:30:52.847225 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 06:30:52.847331 (+ 106us) server_negotiation.cc:407] Connection header received
0806 06:30:52.847380 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:30:52.847381 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:30:52.847390 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:30:52.847442 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:30:52.847763 (+ 321us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56118: BlockingRecv error: recv got EOF from 127.0.0.1:56118 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
W20260806 06:30:52.977581 22591 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 06:30:52.976973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56128 (local address 127.21.12.189:40089)
0806 06:30:52.977037 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0806 06:30:52.977039 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:30:52.977116 (+ 77us) server_negotiation.cc:407] Connection header received
0806 06:30:52.977164 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:30:52.977165 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:30:52.977175 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:30:52.977235 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:30:52.977521 (+ 286us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56128: BlockingRecv error: recv got EOF from 127.0.0.1:56128 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
I20260806 06:30:53.116811 22729 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3179d6681ac24153ab6385dbba22d241 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3179d6681ac24153ab6385dbba22d241" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3179d6681ac24153ab6385dbba22d241" member_type: VOTER last_known_addr { host: "127.21.12.190" port: 44049 } } peers { permanent_uuid: "0350b17ccfb24e5c82c3a2931bbb5caf" member_type: VOTER last_known_addr { host: "127.21.12.189" port: 40089 } } peers { permanent_uuid: "8650c022f46d45c1892693114473bf4b" member_type: VOTER last_known_addr { host: "127.21.12.188" port: 41667 } } }
I20260806 06:30:53.116953 22729 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3179d6681ac24153ab6385dbba22d241 [sys.catalog]: This master's current role is: LEADER
W20260806 06:30:53.235647 22591 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 06:30:53.234997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56142 (local address 127.21.12.189:40089)
0806 06:30:53.235052 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0806 06:30:53.235054 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:30:53.235070 (+ 16us) server_negotiation.cc:407] Connection header received
0806 06:30:53.235151 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:30:53.235152 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:30:53.235162 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:30:53.235209 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0806 06:30:53.235565 (+ 356us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56142: BlockingRecv error: recv got EOF from 127.0.0.1:56142 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
I20260806 06:30:53.600226 22706 catalog_manager.cc:1259] Loaded cluster ID: 022faf664df04188abd49e1508f67d61
I20260806 06:30:53.600301 22706 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 8650c022f46d45c1892693114473bf4b: loading cluster ID for follower catalog manager: success
I20260806 06:30:53.601089 22706 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 8650c022f46d45c1892693114473bf4b: acquiring CA information for follower catalog manager: success
I20260806 06:30:53.601388 22706 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 8650c022f46d45c1892693114473bf4b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260806 06:30:53.682004 22765 catalog_manager.cc:1259] Loaded cluster ID: 022faf664df04188abd49e1508f67d61
I20260806 06:30:53.682085 22765 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 0350b17ccfb24e5c82c3a2931bbb5caf: loading cluster ID for follower catalog manager: success
I20260806 06:30:53.682889 22765 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 0350b17ccfb24e5c82c3a2931bbb5caf: acquiring CA information for follower catalog manager: success
I20260806 06:30:53.683163 22765 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 0350b17ccfb24e5c82c3a2931bbb5caf: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260806 06:30:53.751060 22865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:53.748782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56152 (local address 127.21.12.189:40089)
0806 06:30:53.748892 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0806 06:30:53.748893 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0806 06:30:53.748904 (+ 11us) server_negotiation.cc:407] Connection header received
0806 06:30:53.748931 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:30:53.748932 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:30:53.748985 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:30:53.749059 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 06:30:53.749190 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:53.749691 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:53.750421 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:53.750617 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:53.750662 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:30:53.750776 (+ 114us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 06:30:53.750886 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 06:30:53.750937 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0806 06:30:53.750954 (+ 17us) server_negotiation.cc:299] Negotiation successful
0806 06:30:53.750971 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89,"thread_start_us":55,"threads_started":1}
I20260806 06:30:53.754927 22874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:53.752676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47706 (local address 127.21.12.188:41667)
0806 06:30:53.752789 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0806 06:30:53.752792 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 06:30:53.752800 (+ 8us) server_negotiation.cc:407] Connection header received
0806 06:30:53.752844 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:30:53.752844 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:30:53.752880 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:30:53.752943 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 06:30:53.753047 (+ 104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:53.753520 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:53.754198 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:53.754422 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:53.754476 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:30:53.754608 (+ 132us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 06:30:53.754718 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 06:30:53.754781 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0806 06:30:53.754823 (+ 42us) server_negotiation.cc:299] Negotiation successful
0806 06:30:53.754843 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89,"thread_start_us":56,"threads_started":1}
I20260806 06:30:53.759189 22865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:53.756411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56166 (local address 127.21.12.189:40089)
0806 06:30:53.756453 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0806 06:30:53.756455 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:30:53.756731 (+ 276us) server_negotiation.cc:407] Connection header received
0806 06:30:53.756787 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:30:53.756788 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:30:53.756797 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:30:53.756837 (+ 40us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 06:30:53.756963 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:53.757444 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:53.758468 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:53.758746 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:53.758804 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:30:53.758953 (+ 149us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 06:30:53.758985 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 06:30:53.759013 (+ 28us) server_negotiation.cc:1035] Waiting for connection context
0806 06:30:53.759093 (+ 80us) server_negotiation.cc:299] Negotiation successful
0806 06:30:53.759116 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260806 06:30:53.759294 22874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:53.756662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47708 (local address 127.21.12.188:41667)
0806 06:30:53.756699 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0806 06:30:53.756701 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:30:53.756708 (+ 7us) server_negotiation.cc:407] Connection header received
0806 06:30:53.756725 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:30:53.756726 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:30:53.756733 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:30:53.756775 (+ 42us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 06:30:53.756909 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:53.757603 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:53.758607 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:53.758871 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:53.758926 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:30:53.759031 (+ 105us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 06:30:53.759060 (+ 29us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 06:30:53.759113 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0806 06:30:53.759205 (+ 92us) server_negotiation.cc:299] Negotiation successful
0806 06:30:53.759226 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260806 06:30:53.759598 22884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 06:30:53.756373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35034 (local address 127.21.12.190:44049)
0806 06:30:53.756516 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0806 06:30:53.756518 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0806 06:30:53.756527 (+ 9us) server_negotiation.cc:407] Connection header received
0806 06:30:53.756609 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 06:30:53.756610 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 06:30:53.756671 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 06:30:53.756878 (+ 207us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 06:30:53.757033 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:53.757770 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:53.758554 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 06:30:53.758806 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 06:30:53.758866 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 06:30:53.759031 (+ 165us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 06:30:53.759135 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 06:30:53.759191 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0806 06:30:53.759410 (+ 219us) server_negotiation.cc:299] Negotiation successful
0806 06:30:53.759435 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":52,"threads_started":1}
I20260806 06:30:53.760989 21554 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHhLhT/build/release/bin/kudu with pid 22394
I20260806 06:30:53.770643 21554 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHhLhT/build/release/bin/kudu with pid 22489
I20260806 06:30:53.780421 21554 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHhLhT/build/release/bin/kudu with pid 22597
2026-08-06T06:30:53Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2874 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (5748 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (272740 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST